Chapter 20: Regular Expressions

Section 1

A regular expression is a pattern that describes a set of strings, written in a compact notation that a program can match against text. The simplest patterns are literal: the pattern cat matches the string "cat" wherever it appears, and the pattern hello matches "hello". That sounds trivial, but the power of regular expressions comes from the special characters that extend literal matching: a dot that matches any character, brackets that match one character from a set, and quantifiers that say how many times something may repeat. Together, those pieces let a single short pattern describe a whole family of strings, such as all dates in a certain format, or all words beginning with a certain letter.

Regular expressions are provided in Python by the re module, which is part of the standard library. The module's functions take a pattern string and a text string, and report whether, and where, the pattern matches. Patterns themselves are written as raw strings in Python code — prefixed with r, as in r"\d+" — because raw strings do not process backslashes as escapes, and regular expressions use backslashes heavily. Writing a pattern as an ordinary string would require doubling every backslash, which is error-prone and hard to read, so the raw-string convention is effectively part of the notation.

A pattern is a program in miniature: it is read by a matching engine that tries to fit it against the text, character by character, backtracking when a choice fails. Understanding that the engine tries alternatives in order, and that it prefers certain matches over others, explains much of regular expression behaviour, including the greedy matching covered later in this chapter. For now, the mental model to hold is simple: a pattern describes a shape, and matching asks whether the text contains that shape, and where.

Regular expressions are best used for well-defined, regular patterns in text: formats with fixed structure, such as dates, codes, and identifiers, and searches for words or numbers with known shapes. They are poorly suited to nested or recursive structures, such as markup languages, and to tasks where a simple string method would do, such as checking whether a string starts with a prefix. Choosing regular expressions for the right tasks, and string methods for the rest, is the first skill this chapter teaches, and it matters more than memorising every special character.

import re
print(re.search(r"cat", "the cat sat"))
print(re.search(r"dog", "the cat sat"))

A pattern is a description, not a program that runs on its own: it needs a matching function and a text to work on, and the same pattern can be applied to many texts. Regular expressions describe shapes in text, and matching asks whether a text contains that shape, and where it occurs within the text.

Check: Why are regular expression patterns written as raw strings in Python, and what kinds of text patterns are regular expressions best suited to?

Section 2

The re module provides several functions for matching, and choosing the right one is the first practical decision in any regular expression task. re.search() looks for the pattern anywhere in the text and returns a match object for the first match, or None if there is none. re.match() looks only at the beginning of the text, returning a match only if the pattern matches from the first character. re.fullmatch() requires the pattern to match the entire text, from start to finish, returning None if any part is unmatched. Those three functions differ only in where they look, but that difference determines whether a pattern validates a whole string or merely finds a substring.

A common beginner mistake is to use re.match() where re.fullmatch() is needed, for validation. Suppose a program validates a code with the pattern r"\d{4}", meaning four digits, and uses re.match() on the input "1234abc". The match succeeds, because the first four characters are digits, even though the input as a whole is not a valid code. Using re.fullmatch() instead correctly rejects it, because the trailing letters prevent a full match. Validation almost always means full matching, and the function choice should reflect that: fullmatch() for validation, search() for finding, match() only when a prefix match is genuinely what is wanted.

Match objects, returned by all three functions, carry information about the match: the matched text itself, via .group(), the start and end positions, via .span(), and the captured groups, covered later in this chapter. A match object is truthy, while None is falsy, so the idiom if re.search(pattern, text): tests for a match directly, without needing to compare with None explicitly. That idiom is concise and readable, and it appears throughout code that uses regular expressions.

Another useful function is re.findall(), which returns a list of all non-overlapping matches in the text, rather than just the first. It suits extraction tasks, such as collecting all numbers from a document, where every match matters rather than only the first. Its companion finditer() returns an iterator of match objects, which is better for large texts, since it does not build a full list in memory. Both are covered in more detail later, but knowing they exist helps you choose the right tool from the start: one match, or all matches.

import re
print(re.fullmatch(r"\d{4}", "1234"))
print(re.fullmatch(r"\d{4}", "1234abc"))
print(re.search(r"\d+", "order 42 shipped"))

Choosing search, match, or fullmatch is the first decision in any regex task, and getting it wrong is a common source of validation bugs. A match object is truthy and None is falsy, so an if statement can test for a match directly without an explicit comparison. Findall returns all matches; finditer returns match objects one at a time.

Check: Why does validation require fullmatch() rather than match(), and when would you use findall() instead of search()?

Section 3

Character classes match a single character from a set, and they are written in square brackets. The class [abc] matches one character that is a, b, or c; the class [a-z] matches one lowercase letter, using a range; and the class [0-9] matches one digit. Ranges can be combined, as in [a-zA-Z], which matches any letter, and individual characters can be mixed with ranges, as in [a-zA-Z0-9_], which matches any word character. A class matches exactly one character, no matter how many characters it lists: [abc] matches a or b or c, never ab.

Negation reverses a class: [^abc] matches one character that is not a, b, or c, and [^0-9] matches one non-digit. The caret must be the first character inside the brackets to mean negation; elsewhere, it is a literal caret. Negated classes are useful for matching "everything except" a set, such as all characters except a delimiter, which is a common need when splitting text by a separator without consuming the separator itself.

Several shorthand classes cover the most common sets without brackets. \d matches a digit, \w matches a word character — a letter, a digit, or an underscore — and \s matches whitespace, including spaces, tabs, and newlines. Their negations, \D, \W, and \S, match the complements: a non-digit, a non-word character, and a non-whitespace character. Shorthands are shorter and more readable than the equivalent bracket classes, and they are the forms most programmers use in practice, though the bracket forms remain useful for sets that have no shorthand.

A detail worth knowing is that \w, \d, and \s match Unicode characters by default in Python 3, not just ASCII. \d matches digits from other scripts as well as 0 through 9, and \w matches letters from other alphabets. That is usually what is wanted for text in many languages, but a program that must match only ASCII digits — for example, when validating a code format defined as ASCII digits — should use an explicit class like [0-9] or the re.ASCII flag, which restricts the shorthands to ASCII. Being explicit about which character set a pattern means is part of writing a correct pattern.

import re
print(re.findall(r"[a-z]+", "Hello World 123"))
print(re.findall(r"\d+", "order 42, item 7"))
print(re.findall(r"[^0-9]+", "abc123def"))

Character classes match exactly one character, and their negation matches one character outside the set, which is useful for delimiters. Shorthands like \d, \w, and \s are shorter and more readable than the equivalent bracket classes, and they are what most programmers use in practice. Unicode matching is the default in Python 3; use ASCII classes or flags for ASCII-only needs. Ranges combine freely: [a-zA-Z0-9_] matches any word character. A class lists a set; it still matches one character.

Check: How many characters does a character class match, and why might a program use [0-9] instead of \d?

Section 4

Quantifiers say how many times the preceding element may repeat, and they turn single-character patterns into patterns for words, numbers, and codes. The quantifier * means zero or more repetitions, + means one or more, and ? means zero or one — that is, optional. The pattern \d+ therefore matches one or more digits, which is a number of any length, while \d* matches zero or more digits, which also matches the empty string, a distinction that matters when a match is expected to contain something. The pattern colou?r matches both "color" and "colour", because the u is optional.

Braced quantifiers give exact control: {n} means exactly n repetitions, {n,} means n or more, and {n,m} means between n and m repetitions. The pattern \d{4} matches exactly four digits, which suits a year or a PIN, while \d{2,4} matches two, three, or four digits, which suits a value with a variable but bounded length. Braced quantifiers are the right tool for formats with length rules, because they state the rule directly in the pattern, where it can be read and checked, rather than hiding it in code that measures a match's length afterwards.

Quantifiers apply to the single element immediately before them, which is a frequent source of errors. In the pattern ab+, the + applies to b only, so it matches a followed by one or more bs — "ab", "abb", "abbb" — not one or more repetitions of "ab". To quantify a group of characters, the group must be enclosed in parentheses first, as in (ab)+, which matches "ab", "abab", "ababab". That interaction between quantifiers and groups is fundamental, and it is why groups, covered later, are introduced alongside quantifiers in most treatments.

A further caution is that * and + can match more than intended when the quantified element is broad. The pattern .* matches any sequence of characters, including a very long one, and combined with other pattern parts it can consume text that was meant for a later part of the pattern. That behaviour, called greediness, is the subject of the next section, and it is the single most common cause of surprising matches. For now, the rule to remember is to quantify the narrowest element that expresses the intent: digits for a number, letters for a word, a specific character for a separator.

import re
print(re.findall(r"\d+", "a12 b345"))
print(re.fullmatch(r"\d{4}", "1234"))
print(re.findall(r"(ab)+", "abab ab"))

Braced quantifiers state length rules directly in the pattern, where they can be read and checked, rather than hiding them in later code. Quantifiers apply to the single element before them, so a multi-character sequence must be grouped in parentheses before it can be quantified as a whole. Quantify the narrowest element that expresses the intent.

Check: In the pattern ab+, what does the + apply to, and how do you quantify a multi-character sequence instead?

Section 5

Quantifiers are greedy by default: they match as much as they can while still allowing the rest of the pattern to match. That preference for the longest possible match explains many surprising results. Consider the pattern <.+> applied to the text "<b>bold</b>". A greedy .+ first consumes the entire text after the opening <, then backtracks just enough to let the final > match, which means it matches from the first < to the last >, giving "<b>bold</b>" as one match, rather than the two separate tags a reader might expect. The greedy match is correct according to the pattern's rules, but it is not what most people intend when they write such a pattern.

The remedy is a lazy quantifier, written by adding ? after the quantifier: *?, +?, and ?? match as little as possible while still allowing the rest of the pattern to match. The pattern <.+?> on the same text matches "<b>" first, then "</b>", because the lazy .+? stops at the first > that lets the pattern complete. Lazy quantifiers are the right choice when a pattern should stop at the first possible ending, such as the first closing delimiter, while greedy quantifiers suit cases where the last possible ending is wanted, such as consuming everything up to the final delimiter.

Greediness also interacts with backtracking in ways that affect performance, a topic covered later in this chapter's section on catastrophic backtracking. For now, the practical guidance is to think, for each quantifier, about which ending is intended — first or last — and to choose greedy or lazy accordingly, rather than accepting the default without thought. A pattern whose quantifiers were chosen deliberately is a pattern whose matches are predictable.

A related habit is to prefer a negated class over a lazy dot when the intent is "up to a delimiter". The pattern <[^>]+> matches a tag by consuming characters that are not >, which stops at the first > without any need for laziness, and which cannot overrun the delimiter even in edge cases. Negated classes express "up to but not including this character" directly, and they are often clearer and faster than a lazy dot, which expresses the same idea indirectly through backtracking. Both forms are correct; the negated class is usually the better choice.

import re
text = "<b>bold</b>"
print(re.search(r"<.+>", text).group())
print(re.search(r"<.+?>", text).group())
print(re.search(r"<[^>]+>", text).group())

For each quantifier, decide whether the first or the last possible ending is intended, and choose lazy or greedy accordingly, rather than accepting the default blindly. A negated class expresses 'up to but not including this character' directly, and it is often clearer and faster than a lazy dot for the same purpose. Greediness is the single most common cause of surprising matches.

Check: Why does <.+> match the whole of "<b>bold</b>" as one match, and why is <[^>]+> often a better pattern than <.+?>?

Section 6

Anchors assert a position in the text, rather than matching a character, and they are what turn a pattern that finds a substring into a pattern that validates a position. The anchor ^ asserts the start of the text, or, with the MULTILINE flag, the start of a line, while $ asserts the end of the text, or, with MULTILINE, the end of a line. The pattern ^hello matches "hello" only at the start, and world$ matches "world" only at the end. Anchors consume no characters themselves; they only require that the match occur at the asserted position.

Word boundaries are another kind of anchor. The anchor \b asserts a position between a word character and a non-word character, which is the boundary of a word, and \B asserts a position that is not a word boundary. The pattern \bcat\b matches "cat" as a whole word, but not the "cat" inside "concatenate", because the letters around it are word characters, so no boundary exists there. Word boundaries are the right tool for whole-word searches, which are otherwise hard to express: checking for spaces around a word fails at the start and end of text and beside punctuation, while \b handles all of those cases uniformly.

Anchors combine with fullmatch() in a way worth noting. The function fullmatch() already requires the whole text to match, so anchors inside a fullmatch() pattern are redundant, though harmless. The function search() with anchors, as in re.search(r"^hello", text), is equivalent to re.match(), and the function match() is itself a kind of start anchor. Knowing these equivalences helps you read code that uses different styles for the same intent, and helps you choose the clearest style for your own code: fullmatch() for validation, search() with \b for whole-word finding, and anchors when a position matters within a larger pattern.

A subtlety of $ is that it also matches just before a trailing newline, so the pattern r"abc$" matches the text "abc\n", which can surprise a validator that expects an exact match. fullmatch() does not have that quirk, since it requires the entire text, including any trailing newline, to match, and that is another reason to prefer fullmatch() for validation. When anchors are needed inside a larger pattern, the stricter anchors \A for the absolute start and \Z for the absolute end avoid the newline subtlety entirely.

import re
print(re.search(r"\bcat\b", "concatenate the cat"))
print(re.search(r"^hello", "hello world"))
print(re.fullmatch(r"abc", "abc\n"))

Anchors assert positions, not characters, and they turn substring patterns into position-specific or whole-string patterns. Word boundaries handle whole-word searches uniformly, including at the start and end of text and beside punctuation, where space checks fail. Absolute anchors \A and \Z avoid the trailing-newline subtlety of $.

Check: Why does \bcat\b not match inside "concatenate", and why is fullmatch() safer than $ for validating an exact string?

Section 7

Groups, written in parentheses, serve two purposes: they group pattern elements so that quantifiers and alternation apply to the whole group, and they capture the matched text for later use. The pattern (ab)+ groups "ab" so that + repeats the pair, as noted earlier, and it also captures each match of the group, so that after matching, the program can retrieve what the group matched. A pattern can contain several groups, numbered from left to right by their opening parentheses, starting at 1, with group 0 always meaning the entire match.

Captured groups are retrieved from a match object with .group(n), or all at once with .groups(), which returns a tuple of the groups in order. That makes groups the mechanism for extraction: a pattern that matches a date can capture its year, month, and day as separate groups, and the program can then use those parts individually, for example by converting them to integers and constructing a date object, with validation, as taught in the previous chapter. Extraction through groups is the bridge between pattern matching and the rest of a program's logic.

Groups also interact with alternation, which is covered in the next section, and with backreferences, covered later. For now, the key point is that every pair of parentheses both groups and captures, unless it is explicitly marked as non-capturing, a form covered later in this chapter. That dual role means a pattern written only for grouping — to apply a quantifier to several characters — also captures, which is harmless but can shift group numbers if a new grouping pair is added later. A pattern with several groups should therefore be written with care about numbering, and named groups, covered later, remove that fragility for patterns with many groups.

Nested groups are numbered by their opening parentheses, outermost first, which is a rule that is easy to state but easy to miscount in a complex pattern. A pattern like ((\d+)-(\d+)) has group 1 as the whole "number-number" pair, group 2 as the first number, and group 3 as the second number. Counting opening parentheses from the left gives the numbers reliably, and drawing a pattern's group structure on paper, for a complex pattern, is a habit that prevents numbering mistakes before they become debugging sessions.

import re
m = re.search(r"(\d{4})-(\d{2})-(\d{2})", "date 2026-10-30 here")
print(m.group(0), m.groups())
print(m.group(1), m.group(2), m.group(3))

Every pair of parentheses both groups and captures, unless marked non-capturing, so group numbering must be counted carefully in complex patterns. Nested groups are numbered by their opening parentheses, outermost first, which is easy to state but easy to miscount in a complex pattern. Named groups remove numbering fragility for patterns with many groups. Groups are the mechanism for extraction, not just for grouping.

Check: How are groups numbered when parentheses are nested, and why does adding a grouping pair shift later group numbers?

Section 8

Alternation, written with the vertical bar |, offers a choice between alternatives: the pattern cat|dog matches "cat" or "dog", whichever appears. Alternation has the lowest precedence of any pattern operator, which means it applies to the whole pattern on either side unless parentheses restrict it. The pattern ^cat|dog$ therefore means "cat at the start, or dog at the end", not "cat or dog as a whole", and the intended meaning usually requires parentheses: ^(cat|dog)$ for a whole string that is exactly "cat" or "dog". That precedence rule is the most common source of alternation errors, and parentheses are the remedy.

The order of alternatives matters, because the matching engine tries them from left to right and takes the first that allows the rest of the pattern to match. The pattern cats|cat applied to "cats" matches "cats", since the first alternative succeeds, while cat|cats applied to the same text matches only "cat", because the first alternative succeeds and the engine does not try the longer one. For alternation where one alternative is a prefix of another, the longer alternative should come first, so that it gets its chance before the shorter one consumes the shared prefix.

Alternation inside a group combines with quantifiers and with other pattern parts naturally. The pattern (red|green|blue) matches one colour name, and (red|green|blue)+ would match a sequence of colour names, though without separators that sequence is ambiguous. Alternation is also the mechanism for optional spellings and variants: colou?r uses a quantifier for a single-letter difference, but gray|grey uses alternation for a difference that is not a simple optionality, and centre|center does the same for a transposed pair. Choosing between a quantifier, a character class, and alternation for a variation is a matter of which expresses the variation most directly.

A practical note is that long alternations of literal words can often be replaced by a simpler pattern, such as a character class or a word list checked after matching, and that replacement is worth considering when the alternation grows unwieldy. A pattern with twenty alternatives is hard to read and hard to maintain, while a set of words checked with in is clear and easy to update. Alternation shines for a handful of alternatives in a pattern; for a large vocabulary, a data structure is usually the better tool.

import re
print(re.search(r"cat|cats", "cats").group())
print(re.search(r"cats|cat", "cats").group())
print(re.fullmatch(r"(red|green|blue)", "green"))

Alternation has the lowest precedence, so parentheses are needed to restrict it, and longer alternatives should come before shorter prefixes. For a large vocabulary, a set of words checked with in is clearer and easier to update than a long alternation of literal words. Alternation tries alternatives left to right, taking the first that works. Parentheses restrict alternation to the intended scope.

Check: Why does cat|cats match only "cat" in "cats", and why does ^cat|dog$ not mean a whole string that is "cat" or "dog"?

Section 9

Escape sequences give patterns access to characters that are otherwise special, and to characters that are hard to write literally. A backslash before a special character removes its special meaning: \. matches a literal dot, \* matches a literal asterisk, and \\ matches a literal backslash. That is essential when a pattern must match punctuation that is also a pattern operator, such as a decimal point in a number, a dot in a filename, or parentheses in a mathematical expression. Forgetting to escape a dot is a classic error: the pattern 3.14 matches "3x14" as well as "3.14", because the unescaped dot matches any character, while 3\.14 matches only the decimal number.

Other escapes denote character types, as covered earlier: \d for digits, \w for word characters, \s for whitespace, and their negations. Further escapes match positions, such as \b for a word boundary, and specific characters, such as \n for a newline and \t for a tab. Together, escapes form a compact vocabulary for describing text precisely, and learning them is mostly a matter of recognising the handful that appear constantly — \d, \w, \s, \., \\ — and looking up the rest when needed.

The re.escape() function escapes a string so that it can be used as a literal pattern, escaping every special character it contains. That is the correct way to include user-supplied or variable text in a pattern: rather than inserting the text directly, where its special characters would be interpreted as pattern operators, pass it through re.escape() first, so that it matches literally. That practice is also a security matter, covered later in this chapter, since unescaped user input in a pattern can change what the pattern matches in ways the programmer did not intend.

Escapes also appear in replacement strings, for substitution, where \1 refers to the first captured group, a topic covered in the substitution section. For now, the rule to carry forward is simple: in a pattern, a backslash changes meaning — it either removes specialness from a special character, or adds specialness to an ordinary letter — and when in doubt about whether a character needs escaping, escaping it is safe for punctuation, while re.escape() is the safe choice for variable text.

import re
print(re.search(r"3\.14", "value 3.14 here").group())
print(re.search(r"3.14", "value 3x14 here").group())
print(re.search(re.escape("a+b"), "a+b = c"))

In a pattern, a backslash changes meaning: it removes specialness from punctuation, or adds specialness to an ordinary letter like d, w, or s. Forgetting to escape a dot is a classic error: an unescaped dot matches any character, including characters that are not dots at all. Re.escape() makes variable text match literally, whatever special characters it contains. Escaping punctuation is safe; guessing is not.

Check: Why does the pattern 3.14 match "3x14", and why should variable text be passed through re.escape() before inclusion in a pattern?

Section 10

By default, the dot . matches any character except a newline, which means a pattern with a dot does not cross line boundaries. That default suits most text processing, where lines are natural units, but a pattern that must match across lines — for example, a pattern spanning a multi-line comment or a paragraph — needs the DOTALL flag, which makes the dot match newlines as well. Flags are passed to matching functions as a keyword argument, as in re.search(pattern, text, flags=re.DOTALL), or combined with | when several are needed, as in re.IGNORECASE | re.MULTILINE.

The IGNORECASE flag makes matching case-insensitive, so the pattern hello matches "Hello", "HELLO", and "hello" alike. That is convenient for user-facing searches, where case should not matter, but it should not be used for validation of case-sensitive formats, such as codes where case carries meaning. The MULTILINE flag changes the meaning of ^ and $, making them match at line boundaries rather than only at the text boundaries, which suits patterns applied to multi-line documents line by line. Without it, ^ matches only at the very start, which surprises programmers who expect it to match at every line.

The VERBOSE flag allows patterns to contain whitespace and comments, which are ignored, so that a long pattern can be written across several lines with explanations. That flag is covered in its own section later, but it is worth knowing here that flags are not exotic: they are the normal way to adjust a pattern's behaviour for a task, and they are preferable to contorting a pattern to achieve the same effect. A pattern that needs case-insensitivity should use IGNORECASE, not a character class like [Hh][Ee][Ll][Ll][Oo], which is unreadable by comparison.

Flags can also be written inside a pattern, as (?i) for case-insensitivity at the start, though the keyword-argument form is clearer for most code, since the flag is visible at the call site rather than hidden inside the pattern string. When a pattern is compiled once and reused, flags are given at compile time, a topic covered in the section on compilation. The habit to form is to state flags explicitly, at the point where the pattern is used, so that a reader can see the pattern's behaviour without hunting through its text.

import re
print(re.search(r"hello", "HELLO world", flags=re.IGNORECASE).group())
print(re.search(r"^b", "a\nb", flags=re.MULTILINE).group())
print(re.search(r"a.b", "a\nb", flags=re.DOTALL).group())

Flags adjust a pattern's behaviour for a task, and stating them explicitly at the call site makes that behaviour visible to readers. Inline flags hide behaviour inside the pattern string, while keyword flags show it at the call site where a reader can see it immediately. DOTALL makes the dot match newlines, for patterns spanning lines. Prefer flags over contorted patterns for case and line behaviour.

Check: What does each of IGNORECASE, MULTILINE, and DOTALL change, and why is the keyword-argument form of flags clearer than inline flags?

Section 11

When a pattern is used more than once, compiling it with re.compile() is the right practice. Compilation converts a pattern string into a pattern object, which can then be reused for matching, searching, finding, and substitution, with the same methods as the module-level functions but bound to that pattern. The pattern object also carries its flags, which are given at compile time, so every use of the pattern shares the same flags without restating them at each call site. That consistency is a benefit in itself, beyond any performance gain.

Performance is the usual reason given for compilation, and it is real but often overstated. The re module caches recently used patterns internally, so a module-level call with the same pattern string does not recompile from scratch every time. Compilation's main benefits are therefore readability and organisation rather than speed: a compiled pattern can be given a descriptive name, defined once at the top of a module, and used throughout, which makes the pattern's purpose visible and its definition easy to find. A pattern used in a loop over many texts, however, does benefit measurably from compilation, since the cache lookup is avoided and the pattern object is ready to use.

Naming a compiled pattern well matters. A variable named EMAIL_PATTERN or DATE_PATTERN tells a reader what the pattern is for, while a variable named pat or rx tells them nothing. A compiled pattern defined near the code that uses it, with a comment stating the format it matches and an example of a matching string, is self-documenting in a way that an inline pattern string is not. For patterns that appear in several places, a single named definition also prevents divergence, where two copies of a pattern drift apart as one is updated and the other is forgotten.

Compilation also makes testing easier. A test can import a module's compiled patterns and check them against example strings, without needing to exercise the whole program that uses them. That separation — patterns as named, testable objects, distinct from the logic that applies them — is good design, and it is the reason compilation is recommended even when performance is not a concern. Compile patterns that are reused, name them descriptively, and document the format they match.

import re
DATE_PATTERN = re.compile(r"(\d{4})-(\d{2})-(\d{2})")
m = DATE_PATTERN.search("today is 2026-10-30")
print(m.groups())

A compiled pattern defined at module level, with a comment and an example, is self-documenting in a way an inline pattern string is not. Compile patterns that are reused, name them descriptively, and document the format they match with an example string. A single named definition prevents two copies of a pattern drifting apart. Testing is easier when patterns are named objects, importable and checkable in isolation.

Check: Why is naming a compiled pattern better than using an inline pattern string, even though the module caches patterns internally?

Section 12

Extracting all matches from a text is a different task from finding the first match, and findall() and finditer() serve it. findall() returns a list of all non-overlapping matches, scanning the text from left to right and collecting each match in turn. When the pattern has no groups, each list element is the matched string itself; when it has groups, each element is a tuple of the groups, not the whole match, which is a behaviour that surprises programmers who expect the whole match in both cases. That group behaviour is the reason finditer() is often preferred for extraction with groups: it returns match objects, from which both the whole match and the groups can be retrieved, without ambiguity.

Non-overlapping deserves emphasis. After a match is found, scanning resumes after the end of that match, not after its first character, so overlapping potential matches are not reported. The pattern \d+ on "12345" gives one match, "12345", not five overlapping matches of various lengths, because the first match consumes the whole run of digits. That is the expected behaviour for extraction, where each piece of text should be counted once, but it means a program that needs overlapping matches — a rare requirement — must implement its own scanning, for example by advancing one character at a time and searching from each position.

finditer() returns an iterator, which is better than a list for large texts, since it yields matches one at a time without building a full list in memory. A program that processes a large log file, extracting matches line by line or from a whole document, should prefer finditer() and process each match as it arrives, rather than collecting all matches first and then processing them. That streaming style uses constant memory for the match collection, regardless of how many matches the text contains.

A common extraction pattern combines finditer() with group retrieval: iterate over matches, extract the groups, convert them to the needed types, and validate them, as with dates extracted from text and then validated with the previous chapter's tools. That pipeline — match, extract, convert, validate — is the general shape of text extraction, and it keeps each step separate and testable. Matching finds candidates; validation decides which candidates are real.

import re
text = "orders: 42 and 7"
print(re.findall(r"\d+", text))
for m in re.finditer(r"(\d+)", text):
    print(m.group(1))

Matching finds candidates; validation decides which candidates are real, and the two steps should stay separate in extraction code. Finditer yields matches one at a time, using constant memory for the collection regardless of how many matches a text contains. Overlapping matches are not reported by findall, since scanning resumes after each match. Extraction pipeline: match, extract, convert, validate.

Check: Why does findall() return tuples when a pattern has groups, and why is finditer() better for large texts?

Section 13

Substitution replaces matched text with new text, and re.sub() is the function for it. Given a pattern, a replacement, and a text, sub() returns a new string with every non-overlapping match replaced by the replacement. The replacement can be a plain string, in which case it is used literally except for backreferences, or a function, which is called for each match with the match object and must return the replacement string. The function form is more powerful, since it can compute the replacement from the match, for example by converting a matched number or by looking up a matched word.

Backreferences in a replacement string refer to captured groups: \1 inserts the first group's matched text, \2 the second, and \g<name> inserts a named group's text, a form that is clearer when a pattern has many groups. Backreferences let a substitution rearrange matched text, such as swapping a date from day-month order to month-day order, or wrapping a matched word in markers. They also let a substitution preserve part of the matched text while changing the rest, which is the most common substitution pattern: match a wider context, replace it with a rearrangement that keeps the parts that should survive.

sub() also accepts a count argument, which limits how many replacements are made, counting from the left. With count=1, only the first match is replaced, which suits a task like replacing the first occurrence of a word in a heading. Without a count, all matches are replaced, which is the default and the usual intent for cleaning or normalising text. The companion function subn() works like sub() but returns a tuple of the new string and the number of replacements made, which is useful when a program needs to report how many substitutions occurred, for example in a log message or a user-facing summary.

Substitution should be used with the same caution as matching: a pattern that is too broad will replace text it should not touch, and a replacement that discards matched groups will lose information. Testing a substitution on representative examples, including examples that should not change, is essential, since a substitution that changes too much is harder to notice than one that changes too little, because its output still looks like plausible text.

import re
print(re.sub(r"\d+", "N", "order 42 and 7"))
print(re.sub(r"(\w+)@(\w+)", r"\1 at \2", "mail ada@example"))
print(re.subn(r"\s+", " ", "a  b   c"))

Test substitutions on examples that should not change, since a substitution that changes too much is harder to notice than one that changes too little. Count limits replacements from the left; subn() reports how many replacements were made, which suits logging and summaries. Backreferences in replacements let a substitution rearrange matched text while preserving its parts.

Check: When should a substitution use a function as its replacement rather than a string, and what does subn() return that sub() does not?

Section 14

Splitting text with re.split() divides a string at every match of a pattern, returning the pieces between the matches. That is more flexible than the string method split(), which splits only on a fixed separator or on whitespace, because a pattern can describe a variable separator, such as any run of whitespace, any of several punctuation marks, or a separator with optional surrounding spaces. The pattern \s+ splits on any whitespace run, \s*,\s* splits on commas with optional surrounding spaces, and [,;] splits on either a comma or a semicolon, all in a single call.

A distinctive behaviour of re.split() is that groups in the pattern are preserved in the result: if the pattern contains a capturing group, the text matched by that group is included in the returned list, interleaved with the split pieces. That lets a program split on a separator while keeping the separators themselves, for example splitting a sentence on punctuation while retaining the punctuation marks for later reassembly. If the separators are not wanted, the group should be non-capturing, a form covered later, so that only the pieces are returned. That distinction — capturing groups kept, non-capturing groups dropped — is the key to controlling split() output.

The maxsplit argument limits how many splits are made, counting from the left, so that the remainder of the text stays intact as the final piece. That suits tasks like splitting a line into a first field and the rest, where the rest may itself contain the separator and should not be split further. A common example is splitting a log line into a timestamp and a message, where the message may contain spaces or punctuation that look like separators but must be preserved as a whole.

Splitting with a pattern that can match the empty string is a hazard, since an empty match splits at every position, producing a list of single characters or empty strings that is rarely intended. A pattern used for splitting should always match at least one character, and a program that splits on user-supplied patterns should guard against empty matches. That guard is part of the general caution about patterns from untrusted sources, covered later in this chapter.

import re
print(re.split(r"\s+", "a  b\tc"))
print(re.split(r"([,;])", "a,b;c"))
print(re.split(r"\s+", "a b c d", maxsplit=2))

A splitting pattern should always match at least one character, since an empty match splits at every position and produces useless output. Capturing groups in a split pattern are kept in the result; non-capturing groups are dropped, which controls the output precisely. Maxsplit keeps the remainder intact as the final piece, which suits first-field-and-rest tasks. Splitting on a variable separator is what re.split() adds over the string split() method.

Check: Why does a capturing group in a split() pattern appear in the result, and when is maxsplit useful?

Section 15

A match object is the result of a successful match, and it carries more information than just the matched text. Its .group() method returns the whole match, or a specific group's text when given a number or a name; .groups() returns all groups as a tuple; .groupdict() returns named groups as a dictionary; .span() returns the start and end positions of the whole match, and .span(n) the positions of a group; and .start() and .end() return those positions individually. Together, those methods let a program locate a match precisely, extract its parts, and relate it to the surrounding text.

Positions are particularly useful for reporting and for further processing. A program that finds a pattern in a document and reports it to a user should report where it found it — a line number, a column, or a character offset — and spans provide the offsets from which line and column can be computed. A program that replaces or highlights matches needs spans to know exactly which characters to change. A program that extracts several matches and then sorts or filters them by position uses spans for ordering. In each case, the match object is the bridge between the abstract pattern and the concrete text.

Groups that did not participate in a match return None, which happens with optional groups and with alternation branches that were not taken. A pattern like (a)|(b) matched against "b" gives group 1 as None and group 2 as "b", since only the second alternative participated. Code that reads groups must therefore handle None values, either by checking for them or by providing defaults, rather than assuming every group matched. That handling is easy to forget, and it is a common source of errors in extraction code that works for typical inputs but fails for inputs where an optional part is absent.

Match objects are also immutable in the sense that they describe a completed match; they cannot be modified to represent a different match. That immutability is appropriate, since a match is a fact about a particular text and pattern, not a mutable value. A program that needs a different match runs the pattern again, on the same or a different text, and receives a new match object. Treating match objects as read-only records of a match keeps extraction code straightforward and predictable.

import re
m = re.search(r"(\d+)-(\d+)", "range 10-20 here")
print(m.group(), m.span(), m.groups())
print(m.start(1), m.end(1))

Match objects are read-only records of a completed match; a different match needs a new call, not a modified object. Groups that did not participate return None, so extraction code must handle absent optional parts explicitly. Spans let a program report where a match was found, and highlight or replace exactly those characters.

Check: Why do non-participating groups return None, and what are spans used for beyond retrieving matched text?

Section 16

Named groups give groups readable names instead of numbers, written as (?P<name>...) for the group definition, and retrieved with .group("name") or from .groupdict(). A date pattern with named groups — (?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2}) — lets a program retrieve m.group("year") rather than m.group(1), which is clearer to read and immune to renumbering when a new group is added elsewhere in the pattern. For patterns with more than two or three groups, named groups are a significant readability improvement, and they are the recommended style for extraction patterns.

Named groups also appear in substitutions, where \g<name> inserts a named group's text into a replacement string. That form is clearer than numbered backreferences in a replacement, especially when a pattern has several groups, since a reader can see which part is being inserted without counting parentheses. The two forms can be mixed in one pattern, but a pattern that uses named groups for its important parts and numbered groups for incidental grouping is harder to follow than one that names every group it intends to use, and consistency is worth the small extra typing.

Group names must be valid Python identifiers, and they must be unique within a pattern, since a name maps to a single group. A pattern that reuses a name for two groups is an error at compile time, which is a useful guard: it catches a copy-paste mistake where a group definition was duplicated without renaming. That compile-time check is another advantage of named groups over numbered ones, where a duplicated group is silent and merely shifts later numbers.

A related form is the named backreference, written (?P=name) inside a pattern, which matches the same text as the named group previously matched. That is useful for matching paired delimiters, such as an opening and closing quote of the same kind, where the closing delimiter must match the opening one. Named backreferences are an advanced feature, but they illustrate a general point: names make patterns self-documenting, and self-documenting patterns are easier to write correctly and to maintain.

import re
m = re.search(r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})", "2026-10-30")
print(m.groupdict())
print(re.sub(r"(?P<w>\w+)", r"[\g<w>]", "hello"))

Named groups are the recommended style for extraction patterns with more than two or three groups, for readability and for immunity to renumbering. Group names must be valid identifiers and unique within a pattern, which catches copy-paste duplication at compile time. Named backreferences match the same text as the named group, such as a matching closing quote. Named groups in substitutions use \g<name>, which is clearer than numbered backreferences for many groups. A date pattern with named year, month, and day groups reads far better than numbered groups. Names make patterns self-documenting, which helps both writing and maintenance. Use named groups for extraction patterns.

Check: Why are named groups better than numbered groups for a pattern with several groups, and what does a named backreference match?

Section 17

Non-capturing groups, written (?:...), group pattern elements without capturing them, so they do not appear in .groups(), do not consume a group number, and do not appear in findall() tuples or in split() results. They are the right tool when grouping is needed for a quantifier or for alternation, but the grouped text is not needed afterwards. The pattern (?:ab)+ repeats "ab" without capturing it, and the pattern (?:red|green|blue) offers a colour choice without capturing which colour matched, when only the fact of a match matters and not the specific colour.

Using non-capturing groups by default, and capturing groups only when the captured text is needed, is a good discipline. It keeps group numbers stable when grouping is added for structure, it keeps findall() and split() results clean, and it signals intent to a reader: a capturing group says "this part matters", while a non-capturing group says "this part is structure only". A pattern whose groups all matter is easier to understand than one whose groups are a mix of meaningful captures and incidental grouping, where the reader must work out which is which.

Non-capturing groups also combine with flags in a scoped form, written (?i:...), which applies a flag only within the group. The pattern (?i:hello) world matches "HELLO world" case-insensitively for the first word but case-sensitively for the second, which is occasionally useful for patterns with mixed case requirements. Scoped flags are an advanced feature, but they illustrate the same principle as non-capturing groups: grouping can limit the scope of an effect, whether that effect is capturing or flag application, and limiting scope makes patterns more precise.

A pattern that uses only non-capturing groups has no groups at all, which means findall() returns whole matches and split() returns only pieces, both of which are the simplest behaviours for those functions. When a pattern's purpose is finding or splitting rather than extraction, non-capturing groups are therefore the natural choice, and capturing groups should be reserved for extraction patterns where the captured parts are the point.

import re
print(re.findall(r"(?:ab)+", "abab ab"))
print(re.findall(r"(ab)+", "abab ab"))
print(re.split(r"(?:,|;)", "a,b;c"))

Use non-capturing groups by default, and capturing groups only when the captured text is actually needed afterwards. Scoped flags with (?i:...) apply a flag only within a group, for patterns with mixed case requirements. A pattern with only non-capturing groups gives findall whole matches and split only pieces. Capturing says 'this part matters'; non-capturing says 'this part is structure only'. Limiting scope, whether of capturing or of flags, makes patterns more precise. For finding or splitting rather than extraction, non-capturing groups are the natural choice. Non-capturing groups keep results clean. Structure only, no capture. Group without capturing. Use (?:...) for structure. Done.

Check: When should a group be non-capturing rather than capturing, and what does a non-capturing group do to findall() results?

Section 18

Backreferences match the same text that a group previously matched, rather than matching the group's pattern again. The backreference \1 refers to the first group, \2 to the second, and so on, and a pattern like (\w+)\s+\1 matches a repeated word, such as "the the", because the second part must equal whatever the first group captured, not merely any word. That is a capability that groups alone do not provide: a pattern with two \w+ groups would match any two words, while a backreference requires the second to repeat the first exactly.

Backreferences are useful for a narrow set of tasks: finding doubled words in text, matching paired delimiters where the closing delimiter must equal the opening one, and validating formats where two parts must agree, such as a repeated code. They are not a general tool for all repetition, and they should not be used where a simpler pattern suffices, since they make patterns harder to read and can interact with backtracking in ways that affect performance. A backreference in a pattern is a signal that the pattern's logic is more complex than a straightforward description, and it deserves a comment explaining what repetition it requires and why.

Named backreferences, written (?P=name), provide the same capability with names instead of numbers, and they are clearer for patterns with several groups, for the same reasons named groups are clearer in general. The pattern (?P<q>["'])(?P<word>\w+)(?P=q) matches a quoted word where the closing quote matches the opening quote's kind, single or double, which a pattern with a plain character class for the closing quote could not guarantee. That example shows backreferences at their best: enforcing agreement between two parts of a match that a pattern description alone cannot express.

A limitation of backreferences is that they refer to text, not to patterns, and they can only refer to groups that have already matched. A backreference to a group that has not yet matched, or to an optional group that did not participate, fails or matches the empty string depending on context, which is a subtle behaviour that should be avoided by structuring patterns so that backreferences always follow their groups. Keeping backreferences simple and well-commented avoids the worst of that subtlety.

import re
print(re.search(r"(\w+)\s+\1", "the the cat"))
print(re.search(r"(?P<q>['\"])\w+(?P=q)", "'hello'"))

Backreferences enforce agreement between two parts of a match, which a pattern description alone cannot express, but they deserve a comment explaining why. A backreference refers to text already matched, not to a pattern, and it must follow the group it refers to. Backreferences make patterns harder to read, so use them only where agreement is genuinely required. Doubled words are the classic backreference example, and a good test of understanding.

Check: Why does (\w+)\s+\1 match "the the" but not "the cat", and when are backreferences the right tool rather than a simpler pattern?

Section 19

Lookarounds assert what surrounds a match, without consuming any characters, which lets a pattern require context without including that context in the match. A positive lookahead, written (?=...), asserts that what follows matches, so the pattern \d+(?= dollars) matches a number only when it is followed by " dollars", but the matched text is just the number, not the following words. A negative lookahead, written (?!...), asserts that what follows does not match, so \d+(?! dollars) matches a number only when it is not followed by " dollars". Lookbehinds, (?<=...) and (?<!...), do the same for what precedes a match.

Lookarounds are useful for extraction tasks where the context identifies the value but should not be part of it. Extracting a price from "price: 42 dollars" is cleaner with a lookahead that checks for " dollars" than with a capturing group that includes it and must then be stripped. Lookarounds also express conditions that are awkward with ordinary matching, such as "a word not preceded by a certain prefix", which a negative lookbehind states directly. That directness makes lookarounds valuable despite their slightly unusual syntax.

Lookbehinds have a restriction in Python's re module: they must be fixed-width, meaning the lookbehind pattern must match a known number of characters, not a variable number. A lookbehind like (?<=abc) is allowed, since it is three characters, while (?<=a+) is not, since a+ can match varying lengths. That restriction comes from the matching engine's implementation, and it means some lookbehind ideas must be rephrased, for example by using a capturing group instead, or by matching a wider context and extracting the needed part. Lookaheads have no such restriction, since the engine can try them at a position without needing to know their length in advance.

Lookarounds, like other advanced features, should be used sparingly. A pattern with several lookarounds is hard to read, and a simpler pattern with capturing groups often achieves the same result with clearer code. The guideline is to use a lookaround when the context must be checked but must not be consumed, and to prefer a capturing group when the context is also needed, or when the pattern's readability would suffer from lookaround syntax.

import re
print(re.search(r"\d+(?= dollars)", "cost 42 dollars").group())
print(re.search(r"(?<=price: )\d+", "price: 42").group())
print(re.search(r"\d+(?! dollars)", "cost 42 euros").group())

Use a lookaround when context must be checked but not consumed, and prefer a capturing group when the context is also needed. Lookbehinds must be fixed-width in Python's re module, while lookaheads have no such restriction. Several lookarounds in one pattern are hard to read; prefer capturing groups when readability would suffer. Lookarounds assert context without consuming it, which is their defining property. A negative lookahead states 'not followed by' directly, which ordinary matching states only awkwardly.

Check: What is the difference between a lookahead and a capturing group for checking context, and why must lookbehinds be fixed-width in Python's re?

Section 20

Escaping user-supplied text for inclusion in a pattern is a security and correctness practice, and re.escape() is the tool for it. When a program builds a pattern from a user's search term, for example to highlight that term in a document, inserting the term directly into the pattern would let its special characters act as pattern operators: a search for "a+b" would match "aab", "aaab", and other strings, not just the literal "a+b". Passing the term through re.escape() first escapes every special character, so the pattern matches the term literally, regardless of what characters it contains.

That practice matters for security as well as correctness. A pattern built from untrusted input without escaping can be manipulated to match far more than intended, potentially exposing text that should not match, or, in combination with quantifiers, causing performance problems of the kind covered later in this chapter. Escaping does not solve every problem with user-supplied patterns — a user who is allowed to supply a full pattern, rather than a literal term, is a different case, covered later — but for the common case of a literal term inside a larger pattern, escaping is the complete and correct solution.

The same principle applies to any variable text in a pattern, not just user input. A filename, a configuration value, or a word from a data file may contain characters that are special in patterns, and inserting such text without escaping risks the same misinterpretation. The habit to form is to treat every interpolated value in a pattern as literal unless there is a specific reason to treat it as pattern syntax, and to use re.escape() for the literal case. That habit is analogous to parameterised queries for databases: data and pattern, like data and query, should be kept distinct, with an explicit mechanism for combining them safely.

A related habit is to avoid building patterns by string concatenation where a clearer structure exists. A pattern assembled from many escaped fragments is hard to read, and a pattern built from a list of alternatives joined by | is clearer when the fragments are alternatives. The goal is a pattern whose structure is visible in code, with literal parts escaped and pattern parts explicit, so that a reader can tell at a glance which parts are data and which are syntax.

import re
term = "a+b"
pattern = re.compile(re.escape(term))
print(pattern.search("a+b = c"))
print(pattern.search("aaab"))

Treat every interpolated value in a pattern as literal unless there is a specific reason to treat it as pattern syntax, and escape it with re.escape(). Data and pattern should stay distinct, with an explicit mechanism for combining them safely, just as with database queries. Avoid building patterns by concatenation where a clearer structure, like joined alternatives, exists.

Check: Why does inserting a user's search term into a pattern without escaping change what it matches, and what is the analogy with parameterised database queries?

Section 21

Validating an email address with a regular expression is a famous example of a task that looks simple and is not. A pattern like r"\S+@\S+\.\S+" checks the rough shape — something, an at sign, something, a dot, something — and it rejects strings with no at sign or no dot, which catches the most common typing mistakes. It does not, however, validate an email address fully: it accepts addresses that are invalid under the email standards, and it rejects some addresses that are valid, because the standards for email addresses are far more permissive and complex than a short pattern can express. The honest description of such a pattern is a sanity check, not a validation.

That distinction matters for user experience. A sanity check that rejects a valid address frustrates a user who cannot register with their real address, while a check that accepts an invalid address merely passes the problem downstream, where sending a confirmation message will fail and reveal the error anyway. The robust approach for email is therefore a light sanity check — presence of an at sign, a dot in the domain part, no spaces — followed by confirmation through a message the user must act on, which proves both that the address is well-formed enough to receive mail and that the user controls it. No pattern, however elaborate, can prove control, and an elaborate pattern that tries to prove validity will inevitably reject some real addresses.

The same lesson applies to other formats with complex standards: web addresses, phone numbers, and postal codes all have rules that vary by country and by standard, and a pattern that claims to validate them fully is almost certainly wrong for some cases. A pattern should claim only what it checks: a shape, a length, a character set, a rough structure. For formats where correctness matters, the pattern is a first filter, and a dedicated parser or a confirmation step is the real validation. Stating what a pattern does not check is as important as stating what it does.

For a learner's programs, a simple email sanity pattern is a good exercise, provided it is labelled as such. Write it, test it with valid and invalid examples, and note which valid forms it rejects and which invalid forms it accepts. That exercise teaches both pattern writing and pattern humility, which is the more valuable lesson: a pattern is a tool for describing shapes, not a proof of correctness, and its limits should be known and stated.

import re
sanity = re.compile(r"\S+@\S+\.\S+")
print(bool(sanity.fullmatch("ada@example.com")))
print(bool(sanity.fullmatch("not-an-email")))
print(bool(sanity.fullmatch("ada@exam ple.com")))

A sanity check plus a confirmation message is the robust approach for email, since no pattern can prove that a user controls an address.

Check: Why is an email pattern a sanity check rather than a validation, and what step actually proves that a user controls an email address?

Section 22

Extracting numbers from text is one of the most common regular expression tasks, and it illustrates the pipeline from matching to validation. The pattern \d+ finds runs of digits, and findall() collects them all, but the extracted strings are still strings, not numbers, and a program that needs numbers must convert them with int() or float(), handling conversion errors where the matched text is not a valid number in the expected sense. A pattern for decimal numbers, such as r"\d+\.\d+|\d+", matches a decimal point form or an integer form, with alternation ordered so that the longer decimal form is tried first, as taught earlier about alternation order.

Signs and separators complicate number extraction. A pattern that allows an optional sign, r"-?\d+", matches negative integers as well as positive ones, but it also matches the hyphen in a date or a range, such as the hyphen in "10-20", where it is a separator rather than a sign. Context determines which interpretation is correct, and a pattern alone cannot always decide: extracting numbers from "range 10-20" as 10 and -20 would be wrong, while extracting from "temperature -5" as -5 is right. The program must therefore consider the surrounding text, either through a wider pattern that includes context, or through validation after extraction that rejects numbers in impossible positions.

Thousands separators are another complication. A number written as "1,234" contains a comma that a simple digit pattern splits into "1" and "234", losing the fact that they form one number. A pattern that includes the separator, such as r"\d{1,3}(?:,\d{3})+", matches grouped numbers correctly, but it must then strip the commas before conversion, which is a post-processing step. That two-step process — match with separators, strip them, convert — is typical of extraction: the pattern finds the shape, and code after the pattern turns the shape into a value.

The final step, validation, is what turns extracted candidates into trustworthy data. A number extracted from text should be checked against the program's expectations: is it in a plausible range, does it have the expected number of digits, does it appear where a number of that kind should appear? A pattern that matches any number cannot answer those questions, and a program that skips validation will accept numbers that happen to look right but mean something else entirely. Matching finds; validation decides.

import re
text = "prices: 12, 3.50, and -7"
print(re.findall(r"-?\d+\.\d+|-?\d+", text))
print(re.findall(r"\d{1,3}(?:,\d{3})+", "total 1,234 items"))

Matching finds candidates; validation decides which candidates are real, and the two steps should stay separate in extraction code. A pattern for decimals should try the longer decimal form before the integer form, so alternation order matters for correct extraction. Signs and separators interact: a hyphen can be a negative sign or a range separator, and context decides which reading is correct.

Check: Why does extracting "10-20" as 10 and -20 misread the text, and what post-processing does a grouped number like "1,234" need after matching?

Section 23

Log files are a natural source of text for regular expression practice, because they have a regular structure: each line contains a timestamp, a level, and a message, in a consistent order, with consistent separators. A pattern for a simple log line might capture the date, the time, the level, and the message as separate groups, using named groups for clarity, and a program can then process each line by matching it, extracting the groups, and converting the date and time with the previous chapter's tools. That pipeline — pattern, groups, conversion, validation — is the same extraction pipeline from earlier sections, applied to a realistic format.

A log-parsing program should handle lines that do not match, because real logs contain lines that break the pattern: stack traces, multi-line messages, and lines from older versions of the program with a different format. A line that fails to match should not crash the program, and it should not be silently discarded either, since discarding it loses information that might explain a problem. The honest handling is to count unmatched lines, report them, and, where possible, preserve them in a separate category for human inspection. That handling turns a parsing failure into diagnostic information, rather than into a crash or a silent loss.

Multi-line log entries deserve special attention. An entry that spans several lines, such as an error with a traceback, cannot be matched line by line with a pattern that assumes one line per entry. A program that parses such logs must first group lines into entries — for example, by treating a line that starts with a timestamp as the start of a new entry, and following lines as continuations — and then apply its pattern to each entry as a whole. That grouping step is a reminder that regular expressions operate on strings, and that preparing the right string to match is often half the task.

Performance matters for log parsing, since logs can be large. Compiling the pattern once, processing lines with finditer() or a loop over lines, and avoiding patterns with catastrophic backtracking, covered later, are the practices that keep a log parser fast enough for real files. A parser that recompiles its pattern for every line, or that uses a pathological pattern on every line, will be slow on a file that a well-written parser handles in seconds.

import re
LOG = re.compile(r"(?P<date>\d{4}-\d{2}-\d{2}) (?P<level>\w+): (?P<msg>.*)")
m = LOG.match("2026-10-30 INFO: server started")
print(m.groupdict())

A log parser should count and report unmatched lines, preserving them for human inspection rather than discarding them silently. Grouping lines into entries comes before matching, when log entries span several lines, since a pattern assumes the right string to match.

Check: How should a log parser handle lines that do not match its pattern, and why must multi-line entries be grouped before matching?

Section 24

Cleaning text — normalising whitespace, removing extra punctuation, standardising separators — is a task where regular expressions excel, because cleaning rules are naturally expressed as patterns. Collapsing runs of whitespace into a single space is re.sub(r"\s+", " ", text), which handles spaces, tabs, and newlines uniformly, where a string replacement would need a separate call for each whitespace character. Stripping leading and trailing whitespace from each line of a multi-line text combines splitlines(), a per-line strip, and a join, though a pattern with MULTILINE can also do it in one substitution, with care about line boundaries.

Removing or replacing punctuation follows the same pattern: a character class lists the punctuation to remove, and sub() replaces it, either with nothing or with a space, depending on whether words on either side should stay separate. Removing punctuation without a space joins words that were separated by that punctuation — "well-known" becomes "wellknown" if the hyphen is simply deleted — while replacing with a space keeps them separate but may introduce extra spaces that a whitespace-collapsing step must then clean. Ordering cleaning steps matters: punctuation replacement before whitespace collapsing, so that the collapsing step tidies the spaces the replacement introduced.

Cleaning should be conservative. A cleaning step that removes characters it should have kept destroys information that cannot be recovered, while a cleaning step that keeps characters it should have removed merely leaves the text slightly messy. When in doubt, keep the character, or replace it with a space rather than deleting it, so that word boundaries survive. That conservatism is especially important for text that will be searched or analysed later, since a lost character can change a word's meaning or break a later pattern that expects it.

A final cleaning consideration is that cleaning is lossy by nature, and a program should clean a copy of the text, not the original, when the original might be needed later — for display, for auditing, or for reprocessing with different rules. Keeping the original and deriving cleaned versions from it, rather than overwriting the original in place, is a small discipline that prevents a cleaning mistake from becoming data loss.

import re
text = "hello,   world!\n\nnew\tline"
clean = re.sub(r"\s+", " ", text)
print(clean)
print(re.sub(r"[,!]", "", clean))

Order cleaning steps carefully: punctuation replacement before whitespace collapsing, so the collapsing step tidies spaces the replacement introduced. Cleaning is lossy by nature, so keep the original text and derive cleaned versions from it, rather than overwriting the original in place. When in doubt during cleaning, keep the character, or replace it with a space, so word boundaries survive for later processing. Collapsing whitespace with \s+ handles spaces, tabs, and newlines uniformly, where string replacement would need several calls.

Check: Why should punctuation usually be replaced with a space rather than deleted, and why should cleaning operate on a copy rather than the original?

Section 25

Regular expressions are powerful, but they are not always the right tool, and knowing when to use a string method instead is a mark of good judgement. Checking whether a string starts with a prefix is str.startswith(), which is clearer and faster than a pattern with a ^ anchor. Checking whether a string contains a substring is in, which is clearer than re.search() with a literal pattern. Splitting on a fixed separator is str.split(), which is clearer than re.split() with a literal pattern. Replacing a fixed substring is str.replace(), which is clearer than re.sub() with a literal pattern. In each case, the string method states the intent directly, while the pattern states it indirectly, through a matching engine.

The rule of thumb is that a pattern is justified when the thing being matched varies in a way that a fixed string cannot express: a variable number of digits, a choice between alternatives, a character from a set, a repetition with bounds. When the thing being matched is fixed — a single word, a single separator, a single prefix — a string method is the better choice, because it is simpler, faster, and easier to read. A program that uses patterns for fixed strings is paying the cost of pattern complexity without gaining any of its benefits, and it is harder to maintain, since a reader must parse a pattern to discover that it matches a fixed string.

There is also a performance dimension. String methods are implemented as direct operations on the text, while patterns go through a matching engine with backtracking and group bookkeeping. For a single operation, the difference is negligible, but for a loop over many texts, the difference accumulates, and a string method in a hot loop will outperform a pattern for the same fixed task. That is a secondary consideration — readability matters more than speed for most code — but it reinforces the same choice: simple tasks deserve simple tools.

A useful test for a pattern is to ask whether it could be replaced by a string method without losing anything. If the answer is yes, replace it. If the answer is no, because the pattern expresses variation that a fixed string cannot, then the pattern is justified, and it should be written as clearly as possible, with named groups, non-capturing groups where appropriate, and a comment stating what it matches. That test, applied consistently, keeps a codebase's patterns to the cases where they earn their complexity.

import re
print("hello world".startswith("hello"))
print("cat" in "concatenate")
print("a,b,c".split(","))

A pattern is justified when the thing matched varies in a way a fixed string cannot express, such as digits of variable length or a choice between alternatives.

Check: When is a string method a better choice than a regular expression, and what test decides whether a pattern is justified?

Section 26

Catastrophic backtracking is the performance failure mode of regular expressions, and understanding it is essential for writing patterns that are safe on untrusted or large input. It occurs when a pattern contains nested quantifiers — a quantified group containing a quantified element, such as (a+)+ — applied to a text that almost matches but does not, forcing the engine to try an exponentially growing number of ways to divide the text among the quantifiers before concluding that no match exists. A short input can then take an extremely long time to process, effectively hanging the program, and that hang is a denial-of-service vulnerability when the input comes from an untrusted source.

The classic example is a pattern for a simple format that is written too loosely, such as ^(a+)+$ applied to a string of many as followed by a character that is not a. The engine tries every way to split the as between the inner and outer quantifiers, and the number of splits grows exponentially with the input length, so adding a single character can double the processing time. A pattern that behaves that way on a test input of twenty characters will be unusable on an input of thirty, and the transition from fast to hung is abrupt, which is why catastrophic backtracking is often discovered in production rather than in testing.

Preventing it starts with avoiding nested quantifiers where possible. A pattern like (a+)+ can usually be simplified to a+, which matches the same strings without the nested structure, and a pattern with a quantified group containing an optional element can often be rewritten with a clearer structure that does not nest. When nesting cannot be avoided, the pattern should be tested with near-miss inputs — inputs that almost match but fail at the last character — and with increasing input lengths, to reveal exponential growth before deployment. A pattern whose time doubles with each added character is a pattern that must be rewritten, not deployed.

A further defence is to limit input size before matching, and to avoid applying complex patterns to untrusted input without such a limit. Input size limits do not fix a pathological pattern, but they bound the damage, turning a hang into a rejection. For programs that must accept patterns from users as well as texts, the defence is stronger: user-supplied patterns should be treated as untrusted code, validated, and, where possible, run with limits, a topic covered in the security section later in this chapter.

import re, time
pat = re.compile(r"^(a+)+$")
start = time.monotonic()
print(pat.match("a" * 20 + "b"))
print("elapsed:", round(time.monotonic() - start, 4))

Test patterns with near-miss inputs and increasing lengths, to reveal exponential growth before deployment rather than in production.

Check: Why does a nested quantifier like (a+)+ cause exponential backtracking on a near-miss input, and what test reveals it before deployment?

Section 27

Reusing a compiled pattern is straightforward, but reusing it well involves a few practices that keep pattern-heavy code maintainable. Define compiled patterns at module level, with descriptive names, near the top of the file where they are easy to find, and import them where they are needed, rather than defining the same pattern in several functions. That single definition prevents divergence, makes testing easy, and lets a reader see all of a module's patterns in one place, which is valuable when a format changes and every pattern that matches it must change together.

Pattern reuse also benefits from building patterns from parts. A date pattern's year, month, and day parts can be defined as separate strings and combined into a full pattern, so that a change to the year part propagates to every pattern that uses it. That composition should be done with care about grouping: parts that will be quantified or alternated need their own grouping, and parts that are only for structure should use non-capturing groups, so that combining them does not shift group numbers unexpectedly. Named groups help here too, since a combined pattern's groups are identified by name rather than by position.

Caching is another form of reuse, provided by the re module itself for module-level calls, and by explicit compilation for pattern objects. A program that matches the same pattern in a loop should hold the compiled pattern outside the loop, not recompile it inside, even though the module cache would mitigate the cost. Holding it outside is clearer — it states that the pattern is constant for the loop — and it avoids any doubt about cache behaviour. Clarity and performance point the same way here, which is a sign that the practice is sound.

Finally, reused patterns should be tested as units, with a set of examples that should match and examples that should not, covering the format's edge cases. A pattern test suite is small — a list of strings and expected results — and it catches regressions when a pattern is modified for a new requirement. A pattern that is reused in many places deserves that test suite more, not less, than a pattern used once, because a regression in a reused pattern affects every place that uses it.

import re
YEAR = r"\d{4}"
DATE = re.compile(rf"(?P<year>{YEAR})-(?P<month>\d{{2}})-(?P<day>\d{{2}})")
print(DATE.search("2026-10-30").groupdict())

Build patterns from named parts, so a change to one part propagates to every pattern that uses it, without divergence. Hold a compiled pattern outside a loop over many texts, rather than recompiling it inside, for clarity and for speed. A pattern test suite is small: strings that should match, strings that should not, and edge cases for the format. Done.

Check: Why should a reused pattern be defined once at module level rather than in each function that uses it, and why does a reused pattern deserve a test suite more than a one-off pattern?

Section 28

Flags adjust how a pattern matches, and the most commonly used flags deserve a closer look beyond their introductions. IGNORECASE makes letter matching case-insensitive, which suits user-facing searches and case-insensitive formats, but it also affects character classes and ranges in ways that can surprise: with IGNORECASE, the class [a-z] also matches uppercase letters, since case-insensitivity applies to the class as a whole. A pattern that relies on a class to distinguish cases — for example, a code format where lowercase and uppercase mean different things — must not use IGNORECASE, or it will accept inputs it should reject.

MULTILINE changes ^ and $ to match at line boundaries, which suits patterns applied to multi-line documents where each line should be treated independently. Without MULTILINE, a pattern with ^ matches only at the document's start, which is correct for a pattern that should match the document as a whole, but wrong for a pattern that should match a line within it. The choice between the two depends on whether the unit of matching is the document or the line, and a program that processes a document line by line in a loop usually does not need MULTILINE at all, since each line is matched separately and ^ already means that line's start.

DOTALL makes the dot match newlines, which suits patterns that span lines, such as a pattern for a multi-line block. Without it, a dot stops at a line boundary, which is correct for patterns that should stay within a line. The two flags are sometimes confused, because both concern lines, but they concern different things: MULTILINE concerns anchors, DOTALL concerns the dot, and a pattern can need either, both, or neither, depending on whether it anchors to lines and whether it spans them.

The ASCII flag restricts \d, \w, \s, and related shorthands to ASCII characters, which suits formats defined as ASCII, such as codes and identifiers in systems that accept only ASCII. Without it, those shorthands match Unicode characters, which is correct for natural language text but wrong for an ASCII-only format. Being explicit about ASCII versus Unicode is part of writing a correct pattern for a defined format, and the flag is the mechanism for that explicitness.

import re
print(re.findall(r"[a-z]+", "ABC def", flags=re.IGNORECASE))
print(re.findall(r"\d+", "a1 b2", flags=re.ASCII))
print(re.search(r"^b", "a\nb", flags=re.MULTILINE).group())

ASCII versus Unicode is an explicit choice for defined formats, and the ASCII flag is the mechanism for stating it. MULTILINE concerns anchors and DOTALL concerns the dot; a pattern can need either, both, or neither, depending on its task. Case-insensitivity applies to character classes as a whole, so [a-z] with IGNORECASE also matches uppercase letters. A program processing lines one by one usually needs no MULTILINE flag, since ^ already means each line's start.

Check: How does IGNORECASE change the meaning of [a-z], and what is the difference between what MULTILINE and DOTALL each affect?

Section 29

Long patterns are hard to read, and the VERBOSE flag is the standard remedy. With VERBOSE, whitespace in a pattern is ignored, except inside character classes and when escaped, and # starts a comment that runs to the end of the line. A pattern can therefore be written across several lines, with each part on its own line and a comment explaining it, turning a dense one-line pattern into a readable, documented structure. That readability is worth the extra lines, especially for patterns that will be maintained by someone other than their author, which in practice means the author themselves, months later.

Verbose patterns also make grouping and alternation clearer, since each alternative can occupy its own line, and each group's purpose can be commented. A date pattern in verbose mode can show its year, month, and day parts on separate lines, with comments stating the range each part allows, while the same pattern in compact mode is a single dense string that must be parsed character by character. The verbose form is longer, but length in lines is not complexity: a pattern spread over ten readable lines is simpler to understand than the same pattern compressed into one line.

A caution with VERBOSE is that whitespace inside the pattern is now ignored, so a pattern that must match a literal space must write that space as \ , as [ ], or as \s, rather than as a bare space in the pattern text. Forgetting that rule produces a pattern that matches without the space it was meant to require, which is a subtle bug that verbose mode introduces even as it solves readability. The rule is easy to remember once stated — in verbose mode, spaces must be explicit — but it must be stated, since it contradicts the intuition that a space in a pattern matches a space in the text.

Verbose mode combines well with named groups and with compiled patterns. A compiled, verbose, named-group pattern, defined at module level with a comment, is the most maintainable form a complex pattern can take, and it is the form recommended for any pattern that is long enough to need explanation. Short patterns do not need verbose mode, and using it for them adds noise, but the threshold is low: a pattern with more than two or three groups, or with alternation, usually benefits.

import re
DATE_VERBOSE = re.compile(r"""
    (?P<year>\d{4})   # four-digit year
    -                 # separator
    (?P<month>\d{2})  # two-digit month
    -                 # separator
    (?P<day>\d{2})    # two-digit day
""", flags=re.VERBOSE)
print(DATE_VERBOSE.search("2026-10-30").groupdict())

A compiled, verbose, named-group pattern at module level is the most maintainable form a complex pattern can take. Short patterns need no verbose mode; patterns with several groups or alternation usually benefit from it.

Check: Why does verbose mode require literal spaces to be written explicitly, and when does a pattern benefit enough from verbose mode to justify its extra lines?

Section 30

Common pitfalls in regular expressions form a short list that is worth memorising, because each pitfall has bitten many programmers and each is avoidable with a habit. The first is the unescaped dot, covered earlier: a dot matches any character, so a pattern for a literal dot must escape it. The second is greedy quantifiers consuming too much, covered earlier: a greedy quantifier takes the longest match, so a pattern that should stop at the first delimiter needs a lazy quantifier or a negated class. The third is match() used for validation, covered earlier: match() checks a prefix, so validation needs fullmatch().

The fourth pitfall is alternation precedence, covered earlier: alternation applies to the whole pattern on either side, so parentheses are needed to restrict it. The fifth is group numbering shifting when a grouping pair is added, covered earlier: every capturing group consumes a number, so adding one renumbers later groups, unless named groups or non-capturing groups are used. The sixth is the findall() group behaviour, covered earlier: with groups, findall() returns tuples of groups, not whole matches, so finditer() is often the clearer choice for extraction.

The seventh pitfall is character class confusion: inside brackets, most special characters lose their special meaning, so a dot inside [...] is a literal dot, not any character, while the caret and hyphen have special meanings only in specific positions. A class like [a-z.] matches a lowercase letter or a dot, which is correct, but a class written as [a-z-.] or with a misplaced hyphen can mean something unintended, since a hyphen between two characters denotes a range. Placing a literal hyphen first or last in a class, as in [-a-z] or [a-z-], removes that ambiguity.

The eighth pitfall is forgetting that patterns are strings, and that backslashes in ordinary strings are escapes, which is why raw strings are the convention. A pattern written without the r prefix, such as "\d+", works by accident in that case, since \d is not a recognised Python escape and is left as backslash-d, but a pattern like "\b" does not work, since \b in an ordinary string is a backspace character, not a word boundary. Raw strings remove that entire class of accident, and they should be used for every pattern, without exception.

import re
print(re.search(r"3\.14", "3.14").group())
print(re.search(r"\bword\b", "a word here").group())
print(re.findall(r"[-a-z]+", "well-known"))

Inside brackets, most special characters lose their meaning, while the caret and hyphen are special only in specific positions. Raw strings should be used for every pattern, without exception, since ordinary strings turn some escapes into different characters. Greedy quantifiers, unescaped dots, and match() for validation are the pitfalls to memorise first. A hyphen first or last in a class is literal, removing range ambiguity.

Check: Why does a pattern written without the raw-string prefix fail for \b but appear to work for \d, and where should a literal hyphen go inside a character class?

Section 31

Testing a regular expression is not optional, because patterns fail in ways that are hard to spot by reading them. A pattern test is a small table of strings, each marked as should-match or should-not-match, together with the expected captured groups for the matching cases. Writing that table before or alongside the pattern forces the programmer to state the format's edge cases explicitly: the shortest valid string, the longest valid string, strings that are almost valid but fail on one rule, and strings from neighbouring formats that must not match. A pattern that passes its table is a pattern whose behaviour is known, not merely hoped for.

The table should include adversarial cases, not just typical ones. For a pattern that matches a code of four digits, the table should include "123", "12345", "123a", "a123", "1234 ", and " 1234", each of which tests a different rule: length, character set, and anchoring. For a pattern that matches a word, the table should include the word at the start and end of text, beside punctuation, and inside a longer word, testing word boundaries. Each adversarial case corresponds to a pitfall from the previous section, and a table that covers the pitfalls is a table that catches regressions when the pattern changes.

Testing should also cover performance, for patterns that will run on large or untrusted input. A performance test applies the pattern to a near-miss input of increasing length — a string that almost matches but fails at the end — and checks that the time grows roughly linearly, not exponentially. That test catches catastrophic backtracking before deployment, where it would otherwise appear as a hang under load. Performance tests need not be elaborate: a loop over lengths, with a time measurement, is enough to reveal exponential growth, since the growth is dramatic once it appears.

Finally, pattern tests should live with the pattern, as a test file or as examples in the pattern's documentation, so that a future change to the pattern is checked against the same table. A pattern changed without running its table is a pattern whose behaviour is unknown after the change, and the table exists precisely to prevent that unknown. The discipline is the same as for any code: tests encode intent, and running them verifies that intent survives change.

import re
pat = re.compile(r"\d{4}")
cases = [("1234", True), ("123", False), ("12345", False), ("123a", False)]
for text, expected in cases:
    result = pat.fullmatch(text) is not None
    print(text, result, "OK" if result == expected else "FAIL")

A pattern test table should live with the pattern, so a future change is checked against the same examples. Tests encode intent, and running them verifies that intent survives change.

Check: Why should a pattern test table include almost-valid strings, and what does a performance test with near-miss inputs reveal?

Section 32

Glob patterns and regular expressions both describe sets of filenames or strings, but they are different notations with different rules, and confusing them is a common error. A glob pattern, used by pathlib's glob() and by the glob module from the previous chapters, uses * to mean any sequence of characters, ? to mean any single character, and [...] for a character class, much like a regular expression, but it has no quantifiers, no groups, no alternation, and no anchors: a glob pattern matches a whole filename, not a substring, and its * does not cross directory boundaries in the same way a regular expression's .* crosses characters. A program that needs to select files by name should use globbing, which is designed for that task, while a program that needs to match patterns within file contents should use regular expressions.

Converting between the two notations is occasionally necessary, and the fnmatch module, covered in the standard library chapter, provides glob-style matching for strings, while its translate() function converts a glob pattern into a regular expression. That conversion is useful when a program's matching machinery is built on regular expressions but its users supply glob patterns, as in a file filter with user-supplied patterns. The conversion handles the differences — anchoring the pattern, translating * and ?, and escaping other characters — so that the resulting regular expression behaves like the glob pattern the user wrote.

A related distinction is between glob patterns and shell wildcards, which look identical but behave slightly differently across shells and platforms, particularly regarding hidden files and directory traversal. A program should not rely on shell expansion for its file selection, since that expansion happens before the program runs and varies by shell; instead, it should accept a pattern argument and expand it itself with pathlib globbing, which behaves consistently regardless of the shell. That consistency is worth the small extra code, especially for tools that will run on several systems.

The general lesson is to use the notation designed for the task: globbing for filenames, regular expressions for text patterns, and string methods for fixed strings. Each notation has a scope where it is clear and correct, and using one outside its scope — a regular expression to select files, a glob pattern to parse text — produces code that is harder to read and more likely to be wrong. Choosing the right notation is part of choosing the right tool.

import fnmatch
from pathlib import Path
print(fnmatch.fnmatch("report.txt", "*.txt"))
print(fnmatch.translate("*.txt"))

Globbing suits filenames; regular expressions suit text patterns; string methods suit fixed strings, and each notation has its proper scope. A program should expand patterns itself, consistently, rather than relying on shell behaviour.

Check: Why should a program expand file patterns itself with pathlib globbing rather than relying on shell expansion, and when is fnmatch.translate() useful?

Section 33

Validating an identifier — a username, a code, a slug — is a task where regular expressions are genuinely the right tool, because identifiers have a precise, regular shape: a defined character set, a length range, and rules about which characters may appear where. A username pattern might require that the name start with a letter, contain only letters, digits, and underscores, and be between 3 and 20 characters long, which is expressed as r"[A-Za-z][A-Za-z0-9_]{2,19}" with fullmatch(). That pattern states the rules directly, and it rejects names with spaces, punctuation, or wrong lengths, all in one check.

Identifier validation is also an allowlist: it states which characters are permitted, and rejects everything else, rather than listing forbidden characters and permitting the rest. Allowlists are safer than denylists, because a denylist must anticipate every harmful character, while an allowlist need only state the safe set. A username pattern that allows only letters, digits, and underscores cannot be tricked by a character the programmer forgot to forbid, since any character outside the allowed set is rejected automatically. That allowlist principle applies far beyond identifiers: wherever input must conform to a known shape, stating the allowed shape is safer than listing forbidden shapes.

Length rules deserve care. A pattern with {2,19} after the first character gives a total length of 3 to 20, which is correct but easy to miscount: the first character is counted separately from the quantified rest, so the bounds in the pattern are one less than the total bounds. A pattern written as r"[A-Za-z0-9_]{3,20}" without a first-character rule gives the same total length more directly, but it allows names starting with a digit, which may be undesirable if identifiers must start with a letter for readability or for compatibility with systems that treat leading digits specially. The choice between the two forms depends on whether the first-character rule matters, and the pattern should state whichever rules actually apply, with a comment explaining the counting.

Validation should also be paired with a clear error message. A pattern that rejects a username without explanation frustrates users, who cannot tell whether their name was too short, too long, or contained a forbidden character. A program should therefore check the rules in a way that produces specific messages — for example, by checking length separately from character set, or by reporting the pattern's requirements in the error — rather than relying on a single pattern match to communicate everything. The pattern enforces the rules; the program explains them.

import re
user_pat = re.compile(r"[A-Za-z][A-Za-z0-9_]{2,19}")
for name in ["Ada_1", "1Ada", "Al", "Ada Smith"]:
    print(name, bool(user_pat.fullmatch(name)))

Identifier rules should be enforced with fullmatch(), so a name with extra characters cannot pass by matching only a prefix.

Check: Why is an allowlist pattern safer than a denylist for identifier validation, and why does {2,19} after a first character give a total length of 3 to 20?

Section 34

Extracting key-value pairs from text is a common pattern-matching task, appearing in configuration lines, headers, and simple data formats. A line like name=Ada or host: localhost contains a key, a separator, and a value, and a pattern with named groups can capture the key and value separately: r"(?P<key>\w+)\s*=\s*(?P<value>.*)" for equals-separated pairs, with optional whitespace around the separator. The \s* around the separator handles the common variations in spacing, while the value group captures the rest of the line, which may itself need trimming or further processing.

Values themselves may be quoted, in which case the pattern must handle the quotes: a value like "hello world" should be captured without its quotes, and a pattern that captures the quoted content with a group inside the quotes — r'"(?P<value>[^"]*)"' — does that, using a negated class to stop at the closing quote. That negated-class technique, from the greediness section, is the right tool here, since it cannot overrun the closing quote even when the line contains further quoted values after the first. A lazy dot would also work, but the negated class states the intent more directly: a quoted value contains no quotes.

A line may also contain several pairs, separated by commas or semicolons, in which case finditer() with a pair pattern collects them all, one match per pair, and the program builds a dictionary from the matches. That approach works when pairs are well-formed, but it fails silently on malformed pairs, which simply do not match and are skipped, so a program that uses it should check that the number of pairs found matches expectations, or should report unmatched portions of the line. Silent skipping is a form of silent data loss, and it should be guarded against by verifying completeness after extraction.

Key-value extraction has a limit: it suits flat pairs with simple values, not nested structures or values with complex syntax. A value that itself contains the separator, or a pair format with escaping rules, needs a real parser, not a pattern, and attempting to handle those cases with increasingly complex patterns produces fragile code. The guideline is to use patterns for simple, flat pairs, and to switch to a parser — for example, the configparser module from the standard library chapter, or the JSON tools from the previous chapter — when the format's complexity grows beyond flat pairs.

import re
PAIR = re.compile(r"(?P<key>\w+)\s*=\s*(?P<value>[^,]+)")
line = "name=Ada, age=36"
print({m.group("key"): m.group("value").strip() for m in PAIR.finditer(line)})

Flat pairs suit patterns; nested structures and escaping rules need a real parser, not an increasingly complex pattern. Silent skipping of malformed pairs is silent data loss, and should be guarded by completeness checks. Quoted values need their quotes handled, not just their content captured.

Check: Why is a negated class better than a lazy dot for capturing a quoted value, and why must a program verify completeness when extracting several pairs with finditer()?

Section 35

Substitution with a function as the replacement lets a program compute each replacement from the matched text, which opens up transformations that a fixed replacement string cannot express. A function receives the match object, and it can inspect the groups, convert them, look them up, or decide conditionally what to return. A common use is normalising numbers: a pattern matches numbers in a text, and the function returns each number formatted to a fixed number of decimal places, or with thousands separators, or converted from one unit to another. Another use is redaction: a pattern matches sensitive values, and the function returns a masked version, preserving the value's length or format while hiding its content.

Conditional replacement is a particularly useful form. A function can return different replacements depending on the matched text: for example, replacing a word with its singular or plural form based on a preceding number, or replacing a code with a description looked up in a dictionary, with a fallback for unknown codes. That conditional logic lives in ordinary Python code, inside the function, where it can be tested and debugged like any other code, rather than being contorted into pattern syntax. Keeping logic in code and matching in patterns is a division of labour that makes both parts clearer.

A function replacement also handles cases where the replacement depends on context beyond the match itself, such as a counter that numbers replacements sequentially, or a state that changes as replacements proceed. A closure over a counter, incremented on each call, lets a substitution number its replacements — "item 1", "item 2", and so on — which a fixed replacement string cannot do. That stateful replacement is a powerful technique, but it should be used carefully, since it makes the substitution's output depend on the order of matches, which is left-to-right and deterministic, but which a reader must know to predict the output.

The function form is slower than a string replacement, since it calls Python code for each match, but for most texts that cost is negligible compared with the flexibility gained. A substitution that needs computation, conditionals, or state should use a function; a substitution that replaces every match with the same fixed string should use a string, which is simpler and faster. Choosing between them is another instance of the general rule: use the simplest tool that expresses the intent.

import re
def double(m):
    return str(int(m.group(0)) * 2)
print(re.sub(r"\d+", double, "a 3 b 4"))
print(re.sub(r"\bcat\b", lambda m: "CAT", "a cat and a cat"))

Keep matching in patterns and logic in code: a function replacement lets conditional logic live where it can be tested like ordinary code. A string replacement suits fixed text; a function suits computed replacements.

Check: When does a substitution need a function replacement rather than a string, and why is stateful replacement, such as sequential numbering, order-dependent?

Section 36

Quoted strings appear constantly in text — in code, in configuration, in messages — and extracting them is a pattern task with a few subtleties. The simplest quoted-string pattern, r'"[^"]*"', matches a double-quoted string with no internal quotes, using a negated class to stop at the closing quote. That pattern is correct for strings without escapes, and it is the right starting point. It fails, however, on strings with escaped quotes, such as "say \"hello\"", where the escaped quote inside the string is not a closing quote but the negated class treats it as one, ending the match early and producing a wrong extraction.

Handling escapes requires a pattern that recognises an escape sequence as a unit: a backslash followed by any character counts as part of the string, not as a delimiter. The pattern r'"(?:\\.|[^"\\])*"' expresses that: inside the quotes, each unit is either an escaped character (\\.) or a character that is neither a quote nor a backslash ([^"\\]), repeated as needed. That pattern correctly handles escaped quotes, escaped backslashes, and ordinary characters, and it is a good example of a pattern whose structure mirrors the format's rules: a string is a sequence of units, each of which is either an escape or a non-delimiter.

Single-quoted strings follow the same pattern with single quotes, and a pattern that must handle both kinds needs alternation between the two forms, with a backreference to ensure the closing quote matches the opening kind, as taught in the backreference section. That combination — alternation, backreference, and escape handling — produces a pattern that is correct but noticeably complex, which is a sign that the task is approaching the limit of what patterns handle comfortably. A program that needs to parse a full language's string literals, with all their escape rules, should use a real parser for that language, not a pattern, since the pattern will inevitably miss an edge case of the language's string syntax.

For simple cases — configuration values, log messages, quoted words in prose — the negated-class pattern, with escape handling where needed, is sufficient and appropriate. The lesson is to match the pattern's complexity to the format's complexity, and to recognise when the format has outgrown patterns.

import re
pat = re.compile(r'"(?:\\.|[^"\\])*"')
print(pat.findall(r'say "hi" and "say \"hello\"" done'))

Match a pattern's complexity to the format's complexity, and recognise when the format has outgrown patterns entirely. A string is a sequence of units: escapes or non-delimiters, repeated as needed. Escaped quotes inside a string are content, not delimiters, and patterns must treat them so. For full language string literals, use a real parser, not a pattern. Simple quoted values suit patterns; full string syntax needs a parser.

Check: Why does "[^"]*" fail on a string containing an escaped quote, and what does the (?:\\.|[^"\\])* construction express?

Section 37

Searching files for a pattern — the task performed by tools like grep — is a practical application that combines file reading with pattern matching, and it is straightforward to implement in Python for a learner's own files. The program reads a file line by line, applies a compiled pattern to each line with search(), and prints lines that match, optionally with the filename and line number. That line-by-line approach uses constant memory regardless of file size, since only one line is held at a time, and it is the right structure for searching large files that should not be loaded entirely into memory.

Line numbers and context improve the output. A search tool that prints only matching lines, without saying where they were found, is less useful than one that prints the filename, the line number, and the line itself, so that the user can locate the match in the file. Adding context lines — the lines before and after a match — is a further improvement, though it requires buffering a few lines, which is a small complexity worth adding only when context is genuinely needed. For a learner's tool, filename, line number, and line are sufficient, and they teach the pattern of enriching a match with its location.

Searching several files extends the same loop: iterate over filenames, open each in turn, and search its lines, handling missing or unreadable files with a clear error rather than a crash. A program that searches a directory should use pathlib globbing to select files, as taught in the standard library chapter, and should skip binary files, or handle them gracefully, since applying a text pattern to binary data produces meaningless matches and decoding errors. Checking a file's readability and handling decoding errors explicitly, rather than letting them crash the search, is part of writing a robust tool.

A search tool should also respect the files it searches: it reads them, never writes them, and it should not follow links or traverse directories it was not asked to search, since a search that wanders into unexpected places can be slow and surprising. Those restraints are part of defensive programming for file tools, and they apply to any program that walks a filesystem on a user's behalf. The tool's job is to find patterns in the files it was given, no more and no less.

import re
from pathlib import Path
pat = re.compile(r"TODO")
for path in [Path("notes.txt")]:
    if path.exists():
        for i, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
            if pat.search(line):
                print(f"{path}:{i}: {line}")

A search tool reads files, never writes them, and should not wander into directories it was not asked to search. Filename, line number, and line itself make search output locatable and useful.

Check: Why should a file search read line by line rather than loading the whole file, and what should it print besides the matching line itself?

Section 38

Regular expressions cannot parse nested or recursive structures, and understanding why is important for knowing when to stop using them. A pattern describes a regular language, in the formal sense, which means it can match repetitions and choices but cannot count arbitrary nesting depth. A pattern can match parentheses to a fixed depth — one level, two levels, three levels — by writing out each level explicitly, but it cannot match parentheses to any depth, because that requires remembering how many levels are open, which a pattern's finite matching state cannot do. The same limitation applies to nested markup, nested brackets, and any structure where an opening element must be matched by a corresponding closing element at an arbitrary depth.

That limitation is why the advice "do not parse HTML with regular expressions" is standard. HTML is a nested, recursive structure, with tags that contain other tags, attributes with quoted values, comments, and error recovery rules for malformed documents, none of which a pattern can handle generally. A pattern that extracts simple, flat pieces from HTML — for example, all occurrences of a specific tag's text in well-formed documents — may work for a narrow case, but it will fail on documents that use the format's full complexity, and it will fail silently, producing wrong extractions rather than errors. A real HTML parser, built for the format's grammar, is the correct tool, and it handles nesting, entities, and malformed input as part of its design.

The same reasoning applies to other recursive formats: JSON documents with arbitrary nesting, programming language source code, and mathematical expressions with nested parentheses all require parsers, not patterns. Patterns remain useful within those tasks — for example, to match a single token, such as a number or an identifier, inside a larger parsing process — but the overall structure must be handled by a parser that tracks nesting explicitly, typically with a stack. Recognising that division — patterns for tokens, parsers for structure — is a key design skill.

A practical test for whether a task has outgrown patterns is to ask whether the task requires counting or matching pairs at arbitrary depth. If it does, a parser is needed. If it does not — if the task is flat, with fixed structure and no nesting — a pattern is appropriate. That test, applied early, prevents the common failure mode of a pattern that grows ever more complex in an attempt to handle nesting, becoming unreadable long before it becomes correct.

import re
print(re.findall(r"<b>([^<]*)</b>", "<b>bold</b> and <b>more</b>"))

Patterns suit tokens within parsing; parsers suit overall structure, and recognising that division is a key design skill. A pattern for a flat piece of HTML may work narrowly, but it fails silently on full complexity.

Check: Why can a regular expression not match arbitrarily nested parentheses, and what is the correct tool for parsing HTML instead?

Section 39

Security for regular expressions has two sides: patterns from untrusted sources, and texts from untrusted sources, and both need defences. A pattern from an untrusted source is effectively code, since it controls what the matching engine does, and a hostile pattern can be crafted for catastrophic backtracking, hanging any program that applies it to a suitable text. A program should therefore never compile and run a pattern supplied by an untrusted user without validation, and where user-supplied patterns are a genuine feature, they should be restricted — for example, by limiting pattern length, by forbidding nested quantifiers, or by running matches with a timeout in a separate process, since Python's re module itself provides no timeout.

Texts from untrusted sources are the more common case. A program that applies its own, trusted patterns to untrusted texts must still guard against catastrophic backtracking, since a hostile text can be crafted as a near-miss for a vulnerable pattern, triggering exponential matching time. The defences are the ones covered earlier: avoid nested quantifiers, test patterns with near-miss inputs of increasing length, and limit input size before matching, so that even a vulnerable pattern's worst case is bounded. Limiting input size is a simple, effective guard that belongs at the boundary where untrusted text enters the program.

A further defence is to prefer simple patterns for untrusted input. A pattern that matches a fixed format — digits, letters, a separator — is far less likely to backtrack catastrophically than a pattern with nested groups, alternation, and optional parts. Complexity in a pattern is risk, and a program that processes untrusted text should favour the simplest pattern that expresses its intent, even if a more clever pattern could handle more cases. The cases the simple pattern does not handle can be rejected or handled by other means, which is safer than a clever pattern that hangs.

Finally, a program should never use a regular expression to enforce a security boundary by itself, such as validating a path, a query, or a command, without additional checks. Patterns describe shapes, not meanings, and a shape check can be bypassed by an input that has the right shape but the wrong meaning — a path with the right characters that escapes a directory, for example. Security checks should use dedicated mechanisms — path resolution, parameterised queries, allowlists — with patterns as a first filter, not as the sole guard. That layering is the defensive principle that underlies this chapter's security advice.

import re
SAFE = re.compile(r"[A-Za-z0-9_]{1,32}")
print(bool(SAFE.fullmatch("user_1")))
print(bool(SAFE.fullmatch("../escape")))

Limit input size before matching untrusted text, so even a vulnerable pattern's worst case is bounded by that limit. Simple patterns for untrusted input are safer than clever ones, since complexity in a pattern is risk.

Check: Why is a user-supplied pattern effectively untrusted code, and why should a pattern never be the sole guard for a security boundary?

Section 40

Knowing when not to use a regular expression is as valuable as knowing how to use one, and the cases where a parser or a dedicated tool is better form a recognisable list. Nested structures need parsers, as covered earlier. Formats with a dedicated standard library module — JSON, CSV, configuration files, URLs — should use that module, which handles the format's edge cases, rather than a pattern that reimplements a subset of them incorrectly. A URL pattern, for example, will inevitably miss some valid URLs and accept some invalid ones, while urllib.parse, from the standard library chapter, handles URLs correctly by design.

Formats with escaping rules also favour parsers. A format where a separator can be escaped, or where values can be quoted with nested quotes, has rules that a pattern can approximate but rarely capture fully, and the approximation fails on exactly the inputs where correctness matters most — the inputs that use the escaping rules. A dedicated parser for such a format, or a standard library module that implements it, is the correct choice, and the pattern should be reserved for the simpler sub-tasks within parsing, such as matching individual tokens.

Performance is another reason to choose differently. A pattern applied to a very large text, or in a tight loop over many texts, can be slower than a dedicated algorithm for the same task, such as a string search for a fixed substring, or a specialised parser for a structured format. That performance difference rarely matters for a learner's programs, but it matters for tools that process large volumes of text, and it is worth measuring when a pattern-based solution feels slow, rather than assuming the pattern is the fastest approach.

The decision framework for this chapter can be summarised as a series of questions. Is the thing being matched fixed? Use a string method. Is it a filename pattern? Use globbing. Is it a flat pattern with a regular shape? Use a regular expression. Is it nested, or does it have a dedicated parser? Use the parser. Is it from an untrusted source? Add the security defences from the previous section. Asking those questions, in order, for each text task, leads to the right tool more reliably than reaching for patterns by default, and it is the habit this chapter most wants to instil.

import re
from urllib.parse import urlparse
print(urlparse("https://example.com/path"))
print(re.search(r"https?://\S+", "visit https://example.com/path"))

Ask in order: fixed string, filename, flat pattern, nested or dedicated format, untrusted source, and choose the tool that fits the answer. Dedicated modules handle their formats' edge cases; patterns reimplementing them miss those cases. A URL pattern misses valid URLs and accepts invalid ones; urllib.parse handles URLs by design.

Check: What series of questions decides between a string method, globbing, a regular expression, and a parser for a text task?

Section 41

Times and durations appear in text in several formats, and extracting them is a good exercise in combining the pattern tools from this chapter. A time like "09:30" is two groups of two digits separated by a colon, which the pattern r"(\d{2}):(\d{2})" captures as hours and minutes, while a duration like "2h 30m" is a number followed by a unit letter, captured by r"(\d+)h\s*(\d+)m" for the common hour-minute form. Both patterns are simple, but neither, on its own, guarantees a valid time: "99:99" matches the time pattern's shape while being an impossible time, and a duration pattern cannot tell whether "2h 30m" is sensible in context.

Validation after matching is therefore essential, exactly as with dates in the previous chapter. A matched hour must be between 0 and 23, a matched minute between 0 and 59, and a matched duration's parts must be within plausible ranges for the program's purpose. Converting the captured strings to integers and checking those ranges turns a shape match into a validated value, and rejecting out-of-range values with a clear message is better than carrying an impossible time forward into calculations that would produce nonsense. The pattern finds candidates; the range check decides which candidates are real times.

Times with seconds, or with an AM/PM marker, extend the pattern with optional groups: r"(\d{1,2}):(\d{2})(?::(\d{2}))?\s*([AP]M)?" captures hours, minutes, an optional seconds group, and an optional marker group, where the optional groups return None when absent. Code that reads those groups must handle None for the absent parts, defaulting seconds to zero and treating a missing marker as 24-hour time, or rejecting the marker's absence if the program's format requires it. That handling of optional groups is the same None handling from the match-object section, applied to a realistic format.

A further consideration is ambiguity between times and other colon-separated values, such as ratios ("3:2") or biblical references ("John 3:16"). A pattern that matches any two numbers around a colon will match those too, and a program that extracts times from general text must use context — surrounding words, or a requirement that the match be a plausible time — to distinguish them. Context can come from a wider pattern that includes a preceding label like "at", or from validation that rejects ratios as times when the program's domain makes that distinction. Either way, the pattern alone is not enough, and the program's domain knowledge must complete the job.

import re
m = re.search(r"(\d{2}):(\d{2})", "meeting at 09:30")
h, mi = int(m.group(1)), int(m.group(2))
print(h, mi, 0 <= h <= 23 and 0 <= mi <= 59)

A pattern finds candidates; a range check decides which candidates are real times, and the two steps should stay separate. Context decides ambiguous colons: a label like 'at' distinguishes a time from a ratio.

Check: Why does matching the shape of a time not guarantee a valid time, and how should optional groups like seconds and AM/PM be handled when absent?

Section 42

Extracting dates from text and validating them combines this chapter with the previous one, and it is a realistic pipeline worth building carefully. A pattern for ISO dates, r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})", finds candidates in text, and each candidate's groups are converted to integers and passed to the date constructor, which validates them: the 30th of February raises a ValueError, the 13th month raises a ValueError, and a valid date returns a date object that the program can use for comparison, sorting, and arithmetic. That two-step process — pattern for shape, constructor for validity — is the correct division, since a pattern that tried to enforce month lengths and leap years would be enormously complex and still incomplete.

The pattern should also be careful about word boundaries. Without them, a pattern for dates will match the date-like substring inside a longer number, such as the "2026-10-30" inside "12026-10-301", producing a candidate that is not really a date in the text. Word boundaries, or a requirement that the characters around the match are not digits or hyphens, prevent that over-matching, and they are a good example of context checks that belong in the pattern itself, rather than in later validation. A pattern that matches only standalone dates is a pattern whose candidates are more likely to be real dates.

Candidates that fail validation should be handled explicitly, not silently dropped, when the program's purpose is extraction for a user. A user who wrote a date in a document and finds it missing from the extracted list will be confused, while a program that reports "found a date-like string that is not a valid date, at this position" gives the user a chance to correct it. For a program that extracts dates from untrusted text, silently dropping invalid candidates is acceptable, since the text is not the user's own writing, but the choice should be deliberate, and the program's behaviour should match its purpose.

Sorting and comparing extracted dates uses the date objects, not the strings, since string comparison of ISO dates happens to give chronological order for that format, but relying on it is fragile: a different date format in the text would break the ordering, while date comparison is correct for any format once parsed. Converting to objects at extraction, and working with objects thereafter, is the same discipline as in the previous chapter, applied to dates found by patterns rather than dates read from structured input.

import re
from datetime import date
pat = re.compile(r"\b(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})\b")
for m in pat.finditer("due 2026-10-30 and 2026-02-30"):
    try:
        print(date(int(m.group("year")), int(m.group("month")), int(m.group("day"))))
    except ValueError as e:
        print("invalid date:", m.group(0), e)

Converting candidates with the date constructor validates month lengths and leap years, which no reasonable pattern can enforce alone.

Check: Why should date validation use the date constructor rather than a more complex pattern, and why do date patterns need word boundaries?

Section 43

Rearranging matched text with named groups in substitutions is a clean way to reformat data, because the replacement string names the parts it rearranges rather than numbering them. A date in day-month-year order, "30/10/2026", can be rearranged to ISO order with a pattern that captures day, month, and year as named groups and a replacement that lists them in year-month-day order: re.sub(r"(?P<day>\d{2})/(?P<month>\d{2})/(?P<year>\d{4})", r"\g<year>-\g<month>-\g<day>", text). The named replacement is readable: a reader can see that the year moves first, the month stays middle, and the day moves last, without counting groups.

That readability matters more as patterns grow. A numbered replacement like r"\3-\2-\1" for the same rearrangement requires the reader to count groups to know which part is which, and a change to the pattern that adds a group renumbers the replacement as well, creating a second place to update and a second chance for error. Named groups keep the replacement stable when the pattern's structure changes elsewhere, provided the names themselves do not change, which is a weaker and more manageable dependency than numbering. For any substitution that rearranges more than two parts, named groups are the recommended style.

Rearrangement substitutions should still validate their results. Swapping the parts of a date does not validate the date: "30/02/2026" rearranged to "2026-02-30" is still an invalid date, and a program that rearranges dates should validate them after rearrangement, with the previous chapter's tools, rather than assuming that a successful rearrangement implies a valid date. The substitution changes format; validation checks meaning, and the two steps should not be confused, even when they appear in the same program.

A related use is wrapping matched text, such as surrounding every occurrence of a word with markers for highlighting, which a replacement like r"[\g<word>]" with a named group achieves without disturbing the surrounding text. Wrapping is a safe substitution, since it preserves the matched text exactly and only adds markers around it, but it still requires care about overlapping matches and about words inside longer words, which word boundaries handle. The combination of named groups, word boundaries, and a wrapping replacement is a complete, readable solution for a common highlighting task.

import re
text = "due 30/10/2026"
out = re.sub(r"(?P<day>\d{2})/(?P<month>\d{2})/(?P<year>\d{4})", r"\g<year>-\g<month>-\g<day>", text)
print(out)

Named replacements stay stable when a pattern's structure changes elsewhere, provided the group names themselves do not change. Rearranging a date changes its format, not its validity, so validate after rearrangement with the previous chapter's tools. Wrapping matched text with markers preserves it exactly, adding only the markers around it. A numbered replacement requires counting groups; a named replacement shows which part moves where. Substitution rearranges; validation checks meaning, and the two steps must not be confused. Named groups in replacements: readable, stable, recommended.

Check: Why are named groups better than numbered backreferences in a rearrangement replacement, and why does rearranging a date not validate it?

Section 44

A small library of compiled patterns, collected in one module, is a practical way to organise pattern-heavy code, and it applies the module organisation lessons from earlier chapters to regular expressions. Such a module defines each pattern once, with a descriptive name, a comment stating the format it matches, and an example string, and other modules import the patterns they need rather than defining their own copies. That organisation prevents divergence, makes patterns discoverable, and lets a format change be made in one place, with the pattern's test table updated alongside it.

The module should also define helper functions for common operations on its patterns, such as a function that validates a string against a pattern and returns a clear error, or a function that extracts and converts a match's groups into a typed value, such as a date or a number. Those helpers encapsulate the pattern-plus-validation pipeline, so that callers do not repeat the conversion and validation logic at every use site. A caller that wants to validate a username calls validate_username(name), not USER_PATTERN.fullmatch(name), and the helper's error message explains the failure in terms the user understands, rather than exposing pattern internals.

Patterns in a library should be grouped by domain, not merely listed alphabetically, so that related patterns — dates, times, identifiers, codes — sit together and can be reviewed together. Grouping by domain also reveals duplication: two patterns that match overlapping formats, or a pattern that is a special case of another, can be unified or distinguished explicitly, rather than coexisting as near-duplicates that drift apart. A library review that asks, for each pattern, what format it matches, what it rejects, and what its edge cases are, is a review that improves the whole module's correctness.

Finally, a pattern library should be tested as a whole, with a test table per pattern, run as part of the program's test suite. That testing is the same discipline as for individual patterns, scaled to a module, and it is what makes a library trustworthy: a change to one pattern is checked against its table, and a change that breaks another pattern's table is caught before it reaches users. A library without tests is a collection of guesses; a library with tests is a tool.

import re
USERNAME = re.compile(r"[A-Za-z][A-Za-z0-9_]{2,19}")
ISO_DATE = re.compile(r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})")
def validate_username(name):
    if not USERNAME.fullmatch(name):
        raise ValueError("username must start with a letter, use only letters, digits, underscores, and be 3-20 characters")
    return name
print(validate_username("Ada_1"))

A pattern library's helpers encapsulate pattern-plus-validation, so callers get clear errors rather than pattern internals. Group patterns by domain, so related patterns sit together and duplication becomes visible during review. A library without tests is a collection of guesses; a library with tests is a tool.

Check: Why should a pattern library provide helper functions as well as patterns, and how should its patterns be grouped and tested?

Section 45

Debugging a regular expression is a skill that combines tools and habits, and the most useful habit is to test a pattern against examples incrementally, rather than writing a long pattern in one go and wondering why it fails. Start with the simplest part of the pattern, verify it matches, then add one element at a time, verifying after each addition, so that a failure points to the element just added rather than to the whole pattern. That incremental approach is slower to describe than to do, and it prevents the common debugging session where a long pattern fails and the programmer cannot tell which part is responsible.

The match object's groups and spans are the primary debugging tools. Printing a match's groups shows what each group captured, which reveals grouping errors, such as a group capturing too much because of greediness, or too little because of a misplaced quantifier. Printing spans shows where the match occurred, which reveals anchoring errors, such as a pattern matching a substring when a whole string was intended. A small debugging script that prints a pattern's match, groups, and spans for a set of test strings is often enough to diagnose a pattern problem without any specialised tool.

Online pattern testers, which highlight matches in a text as a pattern is typed, are also useful, though they should be used with care: pasting sensitive text into an online tool sends that text to a third party, so testers should be used only with synthetic or non-sensitive examples. For a learner's exercises, that caution rarely matters, but it is a good habit to form early, since professional work frequently involves sensitive text that must not leave the machine. A local test script, using the same re module the program uses, avoids the issue entirely and tests the exact engine the program will use.

A further debugging habit is to simplify a failing pattern to the smallest case that still fails. If a long pattern fails on a long text, reduce the text to the shortest substring that still fails, and reduce the pattern to the shortest form that still fails on that substring, until the failure is isolated to a single element or interaction. That reduction, known as minimising, is a general debugging technique, and it is especially effective for patterns, where a failure often involves the interaction of two elements rather than either element alone.

import re
pat = re.compile(r"(\d{4})-(\d{2})-(\d{2})")
for text in ["2026-10-30", "2026-13-01", "30-10-2026"]:
    m = pat.search(text)
    print(text, m.groups() if m else None)

Print groups and spans when debugging: groups reveal capture errors, and spans reveal anchoring errors. Minimise a failing case: reduce text and pattern until the failure isolates to a single element or interaction.

Check: Why is incremental pattern building better than writing a long pattern at once, and why should sensitive text not be pasted into an online pattern tester?

Section 46

Hex colours are a small, well-defined format that makes a good pattern exercise, because their rules are precise and their edge cases are instructive. A hex colour in the common six-digit form is a # followed by six hexadecimal digits, which the pattern r"#[0-9A-Fa-f]{6}" expresses directly, with a character class for hexadecimal digits covering both cases. The three-digit shorthand form, # followed by three hexadecimal digits, is a second valid form, handled by alternation: r"#(?:[0-9A-Fa-f]{6}|[0-9A-Fa-f]{3})", with the longer form first, so that a six-digit colour is not matched as a three-digit colour plus extra characters, following the alternation-order rule from earlier.

Validating a hex colour with fullmatch() rejects strings with too many or too few digits, with non-hexadecimal characters, and without the leading #, all of which are common input errors. It does not, however, check whether the colour is sensible for its purpose — a colour that is unreadable against a background, for example — since that is a design judgement, not a format rule. The pattern enforces format; the program's design logic enforces suitability, and the two should not be confused, just as a date pattern enforces shape while the date constructor enforces validity.

Normalising a hex colour is a useful follow-on: converting a three-digit shorthand to its six-digit equivalent by doubling each digit, and converting the result to lowercase, gives a canonical form that can be compared and stored consistently. That normalisation is done with a substitution function or with ordinary string code after matching, not with the pattern itself, since patterns match and extract but do not transform values beyond rearrangement. Keeping transformation in code, after matching, is the same division of labour as elsewhere in this chapter: patterns find, code transforms.

Hex colours also illustrate the value of a pattern library entry: a colour pattern, once written and tested, can be reused wherever colours appear in a program — in configuration, in user input, in generated styles — without rewriting it each time, and its test table, covering valid six-digit and three-digit forms and invalid forms, guards against regressions. A small format, well-handled, is a model for larger formats handled the same way.

import re
HEX = re.compile(r"#(?:[0-9A-Fa-f]{6}|[0-9A-Fa-f]{3})")
for s in ["#ff8800", "#f80", "#ff88", "#gg8800"]:
    print(s, bool(HEX.fullmatch(s)))

A colour pattern enforces format only; whether a colour is readable against a background is a design judgement, not a format rule. Normalise colours after matching, in code, by doubling shorthand digits and lowercasing, for a canonical comparable form. A tested colour pattern, reused wherever colours appear, guards against regressions across a program. Six-digit form first, three-digit second: alternation order prevents partial matches. Hex digits cover both cases: [0-9A-Fa-f] matches uppercase and lowercase alike. A small format, well-handled, models larger formats.

Check: Why must the six-digit alternative come before the three-digit alternative in a hex colour pattern, and what does pattern validation not check about a colour?

Section 47

Whitespace handling in patterns is a frequent source of subtle bugs, because whitespace in text is varied — spaces, tabs, newlines, and other Unicode whitespace — while whitespace in a pattern, especially in verbose mode, has special rules. The shorthand \s matches any whitespace character, which is usually what a pattern intends when it allows separation between parts, while a literal space in a pattern matches only a space, not a tab or a newline. A pattern that allows "a space" between words, written with a literal space, will fail on text separated by tabs, while the same pattern written with \s+ will succeed, and the choice between them should reflect whether the format genuinely allows any whitespace or only spaces.

Line boundaries add another layer. A pattern that processes text line by line, with each line matched separately, does not need to handle newlines at all, since they are removed by the line-splitting step. A pattern applied to a whole multi-line document, by contrast, must decide how newlines interact with its parts: whether the dot should cross them, whether anchors should match at line boundaries, and whether whitespace shorthands should include them. Those decisions are made with the flags from earlier sections — DOTALL, MULTILINE, and the choice of \s versus a literal space — and they should be made explicitly, not left to defaults that happen to work for the test text but fail for real text with different line endings.

Trailing whitespace is a particular nuisance. Text copied from other sources often carries trailing spaces on lines, and a pattern that anchors to the end of a line with $ will match before that trailing whitespace in some configurations, or fail to match at all if the pattern expects the line to end immediately after its last meaningful character. Stripping trailing whitespace from lines before matching, or allowing optional trailing whitespace in the pattern with \s*$, are both reasonable approaches, and the choice depends on whether trailing whitespace is meaningful in the format. For most formats, it is not, and stripping it before matching simplifies every pattern that follows.

The general guidance is to normalise whitespace early — collapsing or stripping it as a cleaning step, as taught earlier — and then to match against the normalised text, rather than writing patterns that must handle every whitespace variation. Normalisation first, matching second, is simpler than matching first with whitespace-tolerant patterns, and it makes the patterns themselves cleaner and easier to test.

import re
print(re.split(r"\s+", "a  b\tc\nd"))
print(re.sub(r"[ \t]+$", "", "line with spaces   ", flags=re.MULTILINE))

A literal space matches only a space, while \s matches any whitespace, and the choice should reflect what the format genuinely allows.

Check: Why does a literal space in a pattern fail on tab-separated text while \s+ succeeds, and why is normalising whitespace before matching often simpler than writing whitespace-tolerant patterns?

Section 48

Unicode handling in patterns is a topic that matters as soon as text contains characters beyond ASCII, which is to say, as soon as text contains real-world names, places, and languages. By default, Python's re shorthands are Unicode-aware: \w matches letters from many alphabets, \d matches digits from many scripts, and \s matches Unicode whitespace. That default is correct for natural language text, where a word may contain accented letters or characters from a non-Latin script, and a pattern that used ASCII-only classes would fail to match such words, or would split them incorrectly at non-ASCII letters.

For defined formats, however, Unicode-awareness can be wrong. A code format defined as ASCII digits should use [0-9] or the ASCII flag, not \d, since \d would accept digits from other scripts that the format does not allow. An identifier format defined for a system that accepts only ASCII should similarly use ASCII classes, while an identifier format for a multilingual system should use Unicode classes deliberately, with the understanding that "letter" then means a letter in any script. The choice should be explicit, documented, and tested with non-ASCII examples in both directions: examples that should match, and examples that should not.

Character classes with explicit ranges also need care with Unicode. A range like [a-z] covers only ASCII lowercase letters, not accented lowercase letters, so a pattern for "lowercase letters" in a multilingual context needs a different approach, such as matching word characters and checking case in code, or using a Unicode-aware property, which Python's re module does not provide directly. That limitation is worth knowing: Python's re is Unicode-aware for its shorthands, but it does not offer Unicode property escapes like "any lowercase letter in any script", so some Unicode-aware tasks require code alongside patterns, rather than patterns alone.

Case-insensitivity with Unicode adds a further subtlety. IGNORECASE matches letters case-insensitively according to Unicode case folding, which handles most cases correctly, but which can produce matches that surprise programmers who think only in ASCII terms, such as matching a character whose uppercase form is a multi-character sequence. For most programs, Unicode case folding is the correct behaviour, and the surprises are edge cases that rarely matter, but a program that validates a case-sensitive ASCII format must not use IGNORECASE at all, as noted earlier, and must test with non-ASCII inputs to confirm its behaviour.

import re
print(re.findall(r"\w+", "café naïve"))
print(re.findall(r"[0-9]+", "a١٢٣ b456"))

Unicode defaults suit natural language, while defined ASCII formats need explicit ASCII classes or the ASCII flag. Python's re has no Unicode property escapes, so some Unicode-aware tasks need code alongside patterns. Test Unicode patterns with non-ASCII examples in both directions: examples that should match, and examples that should not.

Check: When is Unicode-aware matching correct, and when should a pattern restrict itself to ASCII explicitly?

Section 49

Performance tips for regular expressions follow from the engine's behaviour, and they are worth collecting in one place. First, compile patterns that are reused, and hold the compiled pattern outside loops, as taught earlier. Second, prefer string methods for fixed strings, as taught earlier, since they avoid the matching engine entirely. Third, prefer specific character classes and negated classes over broad dots, since a specific class limits how much text a quantifier can consume and reduces backtracking. Fourth, avoid nested quantifiers, as taught in the backtracking section, since they are the primary cause of exponential behaviour. Fifth, anchor patterns where possible, since an anchor lets the engine skip positions where a match cannot start, rather than trying the pattern at every character.

A sixth tip is to fail fast: structure a pattern so that a mismatch is detected early, rather than after consuming a long stretch of text that must then be backtracked. A pattern that starts with a distinctive literal, such as a keyword or a separator, fails quickly on texts that lack that literal, while a pattern that starts with a broad quantifier, such as .*, consumes the whole text before discovering that a later part cannot match, and then backtracks through it. Putting the most distinctive part first is a simple optimisation that often improves performance noticeably, and it also improves readability, since the pattern's identifying feature comes first.

A seventh tip is to limit the text matched, by processing line by line rather than as a whole document where possible, and by limiting input size for untrusted text, as taught in the security section. A pattern applied to a single line cannot backtrack across the whole document, and a pattern applied to a bounded input cannot hang for an unbounded time. Those structural choices — line-by-line processing, input limits — are more effective than micro-optimisations to a pattern's syntax, and they should be made first, before tuning the pattern itself.

Finally, measure rather than guess. A pattern's performance on real data, measured with the time module's monotonic clock or with a profiling tool from a later chapter, is the only reliable guide to whether it needs optimisation, and a pattern that is fast enough on real data should not be contorted for hypothetical speed. Optimisation without measurement risks making a pattern harder to read for no benefit, which is a poor trade.

import re, time
pat = re.compile(r"\bTODO\b")
start = time.monotonic()
for _ in range(1000):
    pat.search("fix this TODO later")
print("elapsed:", round(time.monotonic() - start, 4))

Anchor patterns where possible, so the engine skips positions where a match cannot start, rather than trying every character. Fail fast by putting the most distinctive literal first, so mismatches are detected before long backtracking.

Check: Why does putting a pattern's most distinctive literal first improve both performance and readability, and why should optimisation be measured rather than guessed?

Section 50

This chapter has introduced regular expressions as a tool for describing and matching patterns in text, from literal matching and character classes through quantifiers, groups, and alternation, to substitution, splitting, lookarounds, and the security and performance considerations that govern their safe use. The central ideas are few, and they repay restating: a pattern describes a shape; matching finds that shape in text; groups capture its parts; validation decides whether a candidate is real; and the right tool — string method, glob, pattern, or parser — depends on the task's shape, not on habit. A programmer who holds those ideas can learn any further pattern feature by reading its documentation, since the framework for understanding it is already in place.

The habits this chapter most wants to instil are also few. Write patterns as raw strings, always. Validate with fullmatch(), not match(), when the whole string must conform. Escape variable text with re.escape(). Prefer named groups and non-capturing groups where they clarify intent. Test patterns with a table of should-match and should-not-match examples, including adversarial cases. Avoid nested quantifiers, limit untrusted input, and never let a pattern be the sole guard for a security boundary. Those habits, applied consistently, prevent the majority of pattern bugs and all of the pattern security failures that a learner's programs are likely to encounter.

The next chapter turns to testing, which is the discipline that verifies all the code this book has taught, including pattern code: how to think about test cases, how to structure them, and how to use them to catch regressions when code changes. Pattern tests, as taught here, are a special case of that general discipline, and the next chapter generalises it. Later chapters return to patterns in applied settings — data handling, log analysis, and text processing — where the patterns from this chapter will be used without reintroduction, so the time spent here pays off directly, as with the standard library and date chapters before it.

A good way to consolidate this chapter is to build a small text tool that uses several pattern techniques together: a log searcher that finds lines matching a pattern, extracts fields with named groups, validates them, and reports matches with line numbers, or a text cleaner that normalises whitespace, extracts quoted strings, and rearranges dates. A tool that combines those techniques demonstrates the chapter's lessons in working code, which is a stronger proof of understanding than any single section's Check question, and it is a tool that remains useful long after the chapter is finished.

import re
print(re.findall(r"\b\w+@\w+\.\w+\b", "contact ada@example.com today"))

Raw strings, fullmatch for validation, escaped variables, named groups, tested tables, and bounded input: those habits prevent most pattern bugs.

Check: What are the central habits this chapter wants you to form for writing patterns safely, and why is a combined text tool a stronger proof of understanding than isolated examples?

Practical Task

Build a small log-analysis tool called logscan.py that reads a text log file line by line and reports a summary. It should use a compiled pattern with named groups to match lines of the form YYYY-MM-DD LEVEL: message (LEVEL is one of INFO, WARNING, ERROR), validate each matched date with the datetime module's date constructor (reporting invalid dates with their line numbers instead of silently dropping them), count lines per level, collect all ERROR messages, and print a summary showing the total lines, the count per level, the number of unmatched lines, and the earliest and latest valid dates found. It should also accept a search term from the command line, escaped with re.escape(), and print every line containing that term literally, with its line number. Test it with: a well-formed log, a log with an invalid date, a log with unmatched lines, a search term containing regex special characters (such as a+b), and an empty file. Run the tests and fix any failures before considering the task done.

Chapter Questions

1. Why does validation require fullmatch() rather than match(), and what goes wrong when a validator uses match() on an input with trailing characters?
2. How does greedy matching explain the result of <.+> on "<b>bold</b>", and what are the two remedies for stopping at the first delimiter?
3. What is catastrophic backtracking, why do nested quantifiers cause it, and what three defences protect a program that matches untrusted text?
4. When should a program use a string method, globbing, a regular expression, or a dedicated parser, and what goes wrong when a regular expression is used for a nested format like HTML?
5. Why must variable text be escaped with re.escape() before inclusion in a pattern, and why is a user-supplied pattern itself a security concern rather than just data?