Chapter 26 — Working with APIs and HTTP

Section 1: What an API Is

An API, or application programming interface, is a defined way for one program to ask another program for data or for an action, without needing to know how the other program works internally. Think of a restaurant: you do not go into the kitchen to cook your own meal, you read the menu, place an order with the waiter, and receive the dish. The menu is like an API's documentation, telling you what you can ask for. The waiter is like the API itself, carrying your request to the kitchen and bringing back the result. You do not need to know how the kitchen is organised, only how to order correctly and what to expect in return. That separation is powerful, because it lets programs built by different people, in different languages, work together, as long as they agree on the interface.

Most web APIs today use HTTP, the same protocol that browsers use to fetch web pages, but instead of returning a page for a human to read, they return structured data, usually in JSON format, for a program to process. Your program sends an HTTP request to a specific address, called an endpoint, and the service replies with an HTTP response containing data, or an error explaining why the request could not be fulfilled. That request-response pattern is the core of API work, and learning it well will let you use a vast range of services, from weather data to public datasets to your own future services, all through the same basic pattern, differing only in endpoints, parameters, and data shapes, which are documented per service.

APIs differ from scraping a web page, which means extracting data from HTML meant for humans. Scraping is fragile, because page layouts change frequently and without notice, breaking your extraction code. An API, by contrast, is a deliberate contract: the provider promises that endpoints and data shapes will follow documented rules, and will change only with notice or versioning, as discussed later. That contract makes API-based programs more reliable and more respectful of the provider, because APIs are designed for programmatic access, with rate limits and terms that govern fair use, while scraping may violate a site's terms or overload it with requests it was not designed to handle. When a service offers an API, use it; scraping should be a last resort, and only where it is allowed and appropriate.

Finally, APIs are not only for external services. As you build larger programs, you will design internal APIs: functions and modules with clear interfaces that other parts of your program use, without needing to know their internals. The same principles apply: clear documentation, stable interfaces, predictable errors, and versioning when interfaces change. Learning to use external APIs well therefore teaches you to design good internal ones too, because you learn what makes an interface easy or hard to use, from the consumer's perspective. That dual perspective, as both API user and API designer, is valuable throughout your career, and it begins with understanding APIs as contracts for communication between programs, not as magic or as mere data downloads.

Check: How is using an API different from scraping a web page, and why is an API usually more reliable?

Section 2: HTTP Requests and Responses

HTTP communication follows a simple pattern: a client sends a request, and a server sends back a response. A request contains a method, which says what kind of action is being requested, a URL, which says where the request is going, headers, which carry metadata such as what format the client accepts, and optionally a body, which carries data for actions that create or update something. A response contains a status code, which says whether the request succeeded or failed and why, headers, which carry metadata such as what format the data is in, and optionally a body, which carries the requested data or an error description. Understanding those parts is essential, because every API interaction is built from them, and debugging an API problem usually means examining one of them to see what went wrong.

Methods are verbs that describe the request's intent. The most common is GET, which asks to retrieve data without changing anything on the server. POST asks to create something new, sending data in the body. PUT and PATCH ask to update an existing item, with PUT typically replacing it entirely and PATCH changing only part of it, though providers vary in how strictly they follow that distinction, so always check a specific API's documentation. DELETE asks to remove an item. Used consistently, methods make APIs predictable: a GET should be safe to repeat, because it does not change data, while a POST that creates something should not be repeated blindly, because it might create duplicates. That distinction matters for retries, discussed later.

Status codes are three-digit numbers that summarise a response's outcome, grouped by their first digit. Codes starting with 2 mean success, such as 200 for a successful request. Codes starting with 3 mean redirection, where the client should go elsewhere, though APIs use these less often than browsers do. Codes starting with 4 mean the client made an error, such as 404 for a resource that does not exist, or 400 for a malformed request. Codes starting with 5 mean the server encountered an error, such as 500 for an internal failure, which is not the client's fault but still prevents success. Checking the status code first, before trying to parse a response body, is a fundamental API habit, because it tells you whether the body contains expected data or an error description. Parsing an error body as if it were data is a common bug that produces confusing failures downstream.

Headers, though less visible than bodies, carry important information. Request headers can say what format the client accepts or provide authentication, which must be handled securely and never hard-coded in shared code. Response headers can say what format the data is in, how large it is, whether it can be cached, and how many requests remain before a rate limit is hit. That information helps a well-designed client behave correctly, for example by respecting rate limits or handling cached data appropriately. Headers are metadata, not data, but they affect how data should be interpreted. Ignoring them entirely, while focusing only on bodies, misses information that can prevent errors, such as a content type that differs from what was expected, indicating that the response is not in the format your code assumes and should not be parsed as if it were.

Check: What are the parts of an HTTP request and an HTTP response, and why should you check status before parsing the body?

Section 3: URLs, Endpoints and Paths

A URL for an API request has several parts, and understanding them helps you construct requests correctly and debug them when they fail. The scheme, usually https for secure API communication, says how to connect. The host, such as api.example.com, says which server to contact. The path, such as /users/123, says which resource is being requested, often with identifiers embedded in it, like a user number. The query string, after a question mark, carries optional parameters, such as ?page=2, which modify the request without changing the resource itself, for example by asking for a specific page of results. Together, those parts form an endpoint, a specific address for a specific kind of request. API documentation lists endpoints with their paths, methods, parameters, and expected responses, forming a catalogue you consult when building a client. Paths are often designed to reflect resources, in a RESTful style, where a collection, such as /users, represents many items, and an individual item, such as /users/123, represents one item within that collection, with the identifier distinguishing it. That resource-oriented design makes APIs intuitive once you learn it: to list items, request the collection; to get one item, request its individual path; to create an item, send data to the collection; to update or delete an item, act on its individual path. Not all APIs follow REST strictly. Some use action-oriented paths or a single endpoint for many operations, but resource-oriented paths are common enough that recognising them helps you navigate unfamiliar APIs quickly, by guessing, then verifying from documentation, what a path likely does, rather than treating each endpoint as an unrelated address with no pattern. Query parameters deserve careful attention, because they are easy to get wrong in ways that produce subtle failures. A misspelled parameter name, a value not encoded correctly for special characters, or a required parameter omitted can all cause a request to return different data than expected, or an error, without an obvious cause, because the URL looks almost right. Encoding matters especially for values containing spaces, punctuation, or non-English characters, which must be represented in a URL-safe form. HTTP client libraries usually handle that automatically when you pass parameters as data, rather than building the URL string manually, which is error-prone. Prefer passing parameters through a client's parameter mechanism. It handles encoding correctly and prevents mistakes like missing separators or double question marks, which manual construction often introduces. Finally, treat URLs as data that should be constructed carefully and logged safely. Log endpoints and paths when debugging, so you can see exactly what was requested, but do not log full URLs that contain sensitive information, such as tokens in query strings, because logs can be shared and archived, and sensitive data in logs becomes a security risk. If an API uses tokens in URLs, which is poor practice but still occurs, be especially careful to redact them from logs, or to avoid logging full URLs altogether, logging only the path and method, without query values that might be sensitive.

Check: Why should you pass query parameters through a client library's parameter mechanism rather than building the URL string manually?

Section 4: Status Codes in Detail

Status codes deserve a closer look, because handling them correctly distinguishes a robust API client from a fragile one that treats every response as success. Success codes in the 200 range include 200, meaning the request succeeded and the response contains data, and 201, meaning a new resource was created successfully, often in response to a POST, with the new resource's location sometimes given in a header. A client that checks only for 200, and treats 201 as failure, would wrongly report a successful creation as an error. Handling should therefore accept the range of success codes appropriate to the method, not just a single code, unless the API documentation specifies exactly which code to expect for a given endpoint. Even then, be aware that providers sometimes use slightly different codes for similar outcomes, and a client that is too strict may fail on a technically successful response that uses a different success code than expected. Client error codes in the 400 range indicate problems with the request itself, and each code suggests a different fix. A 400 code means the request was malformed, such as invalid JSON in the body or a missing required field, and the fix is to correct the request, not to retry it unchanged, because retrying a malformed request will fail again. A 401 code means authentication is required or has failed, suggesting credentials are missing, expired, or incorrect, and the fix involves authentication, not request correction. A 403 code means the client is authenticated but not allowed to perform the requested action, suggesting a permissions problem, not a credentials problem, and retrying will not help unless permissions change. A 404 code means the requested resource does not exist, suggesting a wrong identifier, a deleted resource, or a wrong path. Distinguishing those codes, rather than treating all 400-range errors as generic failures, lets a client give specific, actionable feedback, such as telling a user that a resource was not found, rather than merely that something went wrong. Server error codes in the 500 range indicate problems on the server side, such as 500 for an internal error, 502 for a bad gateway, 503 for a service temporarily unavailable, and 504 for a gateway timeout. All of these suggest that retrying later, after a delay, might succeed, because the problem is not with the request but with the server's current state, which may recover. That distinction, retry server errors but not client errors, is a basic rule for robust clients. Finally, error responses often contain a body with additional details, such as a message explaining the error, a code specific to the API, or suggestions for fixing it. A good client should read that body, when present, and include its message in logs or in user-facing errors, rather than discarding it and reporting only the status code, because the body often explains the specific problem, such as which field was invalid or why authentication failed, in a way the generic status code alone does not. That body, however, should be handled carefully, because its format may differ from success responses. Parsing it as if it were success data, without checking status first, is a bug.

Check: Why should a client retry a 500-range error but not a 400-range error unchanged?

Section 5: Headers and Content Types

Headers carry metadata about a request or response, and content type is one of the most important headers, because it tells the recipient how to interpret the body. A request that sends JSON data should declare that its body is JSON, through its content type header, so the server knows to parse it as JSON, rather than as form data or plain text, which would fail or misinterpret it. A response's content type header tells the client what format the body is in, such as JSON, so the client knows to parse it as JSON, rather than assuming JSON and failing if the server returns something else, such as an HTML error page. That can happen when a request goes to the wrong URL or when a server is misconfigured. If that HTML is parsed as JSON, it will fail with a confusing error that hides the real problem. Checking content type first would reveal immediately that the response is HTML, not JSON, pointing at a wrong URL or a server issue, rather than at a JSON parsing bug in the client. Other headers also matter for API clients. An accept header tells the server what formats the client can handle, allowing the server to choose an appropriate format. Most APIs return JSON by default and may ignore accept headers, but setting it explicitly is good practice, because it documents the client's expectation and can help diagnose format mismatches, where the server returns a format the client did not expect, indicating a configuration or documentation issue. Authentication headers carry credentials, in a standardised form, such as a token, and they must be handled securely: never logged in full, never hard-coded in shared code, and preferably managed through secure, approved mechanisms. Copying example code that includes a placeholder token, then replacing the placeholder with a real token without considering where that code will be shared, is a mistake that has caused many real security incidents, where credentials leaked through public code repositories because they were hard-coded and then published. Secure handling, through environment configuration or secure vaults where appropriate, and through never committing credentials, prevents that. Rate limit headers, where provided, tell a client how many requests it can make, how many remain, and when the limit resets. Respecting limits is both polite and practical, because it prevents avoidable errors and blocks. It also shows the client is a good citizen of the API ecosystem, which providers appreciate. Finally, where headers provide a reset time for a rate limit, a client should wait until that time rather than guessing, because guessing either retries too soon, causing another limit error, or waits too long, wasting time. Header-guided waiting is efficient, not just polite, and should be implemented whenever headers provide the necessary information, rather than ignoring headers and guessing, which is inefficient and unreliable for any client that makes more than a few requests. In general, reading headers is part of reading a response properly: status tells you whether the request succeeded, headers tell you how to interpret the body and how to behave next, and the body carries the data itself. A client that reads all three is far more robust than one that reads only the body.

Check: Why should you check a response's content type before parsing its body as JSON?

Section 6: Query Parameters and Filtering

Query parameters let a client ask an API to filter, sort, paginate, or otherwise modify the data it returns, without changing the endpoint's path, by appending key-value pairs to the URL's query string. For example, a parameter might ask for only items in a certain category, or for items sorted by date, or for a specific page of results, allowing the server to do work that would otherwise require the client to download all data and filter it locally. That local approach would be inefficient, especially for large datasets, where downloading everything to find a few relevant items wastes bandwidth, time, and memory, while server-side filtering returns only the relevant subset, making the client faster and lighter. The server can also use indexes and optimised queries, rather than sending all data for the client to filter. That division of labour is central to API design: the server, which holds the data, should do filtering, sorting, and pagination, where possible, and the client should request exactly what it needs, through parameters. Server-side parameters also make intent visible. Requesting a category through a parameter makes the client's intent explicit in the request itself, whereas downloading all categories and filtering locally hides intent in client code. That visibility helps debugging, because a request's parameters, logged safely, show exactly what was asked for, making it easy to see whether a wrong filter, a wrong sort, or a wrong page caused unexpected results. Local filtering, hidden in code, requires reading code to understand what was requested, which is less transparent, especially when debugging a live issue where logs are the main evidence. Parameters in logs provide that evidence directly, without needing code. That transparency benefit, in addition to the efficiency benefit, makes server-side parameters preferable whenever an API supports them, which most well-designed APIs do, for common operations like filtering, sorting, searching, and pagination. There is a judgement to make, however. Local processing is simpler to code, and for a small dataset it may still be efficient enough. A beginner pattern is to download everything and then filter, sort, and paginate locally, because it is easier to write. That pattern becomes a performance problem as data grows, and should be replaced, as soon as the API's parameters are understood, with server-side requests that return only needed data. The replacement improves performance, reduces bandwidth, and makes intent visible, all at once. So the rule is: use server-side parameters where the API provides them, especially for large data, and reserve local processing for cases where the API does not support the needed operation, or where the dataset is small enough that local processing is simpler and still efficient. Make that judgement per operation, based on API support and dataset size, rather than defaulting to local processing for everything. Finally, remember that parameters are part of the request, and they should be constructed through a client library's parameter mechanism, as discussed earlier, so that values are encoded correctly. A parameter value containing spaces or punctuation, if built by hand into a URL string, may break the request or be misinterpreted. Passing parameters as data to the library avoids that.

Check: Why is server-side filtering through query parameters better than downloading all data and filtering locally?

Section 7: Sending Data with POST and Bodies

Sending data to an API, typically with POST to create something, or with PUT or PATCH to update something, involves including that data in the request body, usually as JSON, with the content type header declaring it as JSON, so the server knows how to parse it. Some APIs are lenient and ignore unknown fields, while others are strict and reject them, a variation that documentation should clarify, and that clients should respect, by sending only documented fields, with correct types, rather than sending extra fields in the hope the server will ignore them. Validation should happen on both sides. Client-side validation, before sending, gives immediate feedback without a network round trip, and can explain problems in plain language tailored to the user's context, such as which field in a form was invalid and why. Server-side validation remains essential as a second layer, because client validation alone cannot guarantee correctness, especially when data comes from users or from other systems, where client validation may be bypassed or buggy. Server validation provides a safety net that catches invalid data before it is stored, ensuring invalid data cannot be saved even if client validation is bypassed. That layered validation, client plus server, is standard for robust APIs: client validation for user experience, giving immediate, clear feedback, and server validation for security and integrity, as the authoritative check. Neither alone is sufficient. Error handling for data-sending methods deserves as much attention as success handling, because failures are common, due to validation, permissions, or network issues. A good client translates server errors into user-friendly messages, where possible, rather than showing raw server errors to users, who may not understand them. That translation requires reading server error bodies and mapping their messages to user guidance, so users know how to fix the problem, whether it is a missing field, an invalid value, or a permissions issue, rather than merely knowing that something failed, which is unhelpful and frustrating. Finally, do not assume a response body is always present. A successful DELETE, for example, may return only a status, with no body, or a success with a short confirmation, depending on the API. A client that expects a body that a successful DELETE does not return will fail with a parsing error, even though the deletion succeeded, a false failure that status-first handling avoids, by treating a success status as success, even without a body. The same applies to some updates, where the server returns only a status, not the updated resource. Documentation should state that, and clients should handle it by checking status and then parsing a body only if present and expected. That defensive pattern makes clients robust to API variations in response bodies, which, like request bodies, vary by API and by endpoint. Flexible, status-first, content-type-aware response handling is essential for any client that uses multiple endpoints or multiple APIs, where response shapes vary, and where rigid assumptions will break, while flexible handling will adapt, by reading what the response actually contains, rather than what the client assumed it would contain.

Check: Why should you validate data on the client before sending it, even though the server will also validate it?

Section 8: Authentication Concepts

Authentication is how an API verifies that a client is who it claims to be, and it is essential for APIs that protect data or actions, such as those that access a user's private information, or that allow creating, updating, or deleting data on a user's behalf. Without authentication, anyone could read private data or modify it maliciously, a security risk that authentication prevents, by requiring clients to prove their identity, through credentials, such as a token, a key, or a username and password, though the specific mechanism varies by API, and should be learned from its documentation, rather than assumed. Different APIs use different authentication methods, with different security properties, and using the wrong method, or using the right method incorrectly, such as sending credentials in a URL, where they may be logged, or hard-coding them in code, where they may be exposed through sharing, can undermine authentication's security, even if the mechanism itself is sound. Security depends not just on the mechanism, but on how credentials are stored, transmitted, and handled, all of which must be done securely, through approved, secure methods, such as secure vaults for storage, encrypted transport through HTTPS for transmission, and never logging credentials in full, in logs, in error messages, or in shared files, where they could be exposed. Secure habits, such as using placeholders for examples, using secure storage for real credentials, and reviewing code and logs for credentials before sharing, prevent that, by making credential exposure visible and avoidable, rather than accidental and hidden, until it is too late. Once a credential has been exposed, removal alone is not enough, because exposure cannot be undone by deletion once it has been seen by others. The credential must be revoked, as discussed for secrets in version control, a lesson that applies to API authentication as to all secrets, where prevention, through secure handling from the start, is far better than cure, through revocation after exposure, which, though necessary, is disruptive and may not fully mitigate harm, if the exposed credential was already misused. That rule should be maintained rigorously: real credentials kept only in secure storage, accessed through approved mechanisms, and never appearing in code, in logs, in documentation examples, or in shared files, under any circumstances. Followed consistently, it prevents the most common API security failures, which are not sophisticated attacks, but simple exposures through careless handling, that attackers exploit by searching public code, logs, or files for leaked credentials, a search that is automated and constant, making any exposed credential likely to be found quickly, and misused, unless revoked promptly. Secure handling, from the start, eliminates that risk, by ensuring credentials never appear in searchable places. It requires discipline, to resist the temptation to paste a credential for quick testing, and instead to use secure, approved methods, even when they take a little longer to set up, because the time saved by pasting is trivial compared with the time and harm caused by a leak. Once set up, secure storage and test mechanisms often become easier than pasting, and secure handling becomes routine, not exceptional, for every API you use.

Check: Why must API credentials never appear in code, logs, or shared files, even for testing?

Section 9: Handling API Errors Gracefully

Handling API errors gracefully means responding to failures in a way that is helpful, safe, and recoverable, rather than crashing, showing cryptic messages, or losing user data. It requires distinguishing error types, as discussed for status codes, and responding appropriately to each, with user-friendly messages for users, detailed logs for developers, and, where appropriate, retries, fallbacks, or safe defaults, that allow the program to continue, or to fail safely, without corrupting data or misleading users into thinking an operation succeeded when it failed. A false success is worse than a clear failure, because it causes users to act on incorrect assumptions, such as believing data was saved when it was not, or believing a payment was processed when it was not, consequences that can be serious, depending on the operation. Graceful error handling prevents that, by ensuring failures are visible, understood, and, where possible, recoverable, rather than hidden, confusing, or destructive. That visibility starts with checking status and reading error bodies, to understand the specific failure, rather than treating all failures identically, with a generic message, which gives users no guidance on how to fix the problem, or whether retrying will help. Both generic approaches are wrong for some error types, and lead to either giving up too easily, when retry would succeed, or retrying futilely, when correction is needed, both of which waste time and frustrate users. Error-specific handling, by contrast, responds correctly to each error type, maximising the chance of recovery, while minimising wasted retries, an efficiency and user experience benefit that makes it worth the extra code, compared with generic handling, which, though simpler to write, is less helpful and less efficient in practice. Error handling also matters for data safety. A failed remote operation must not leave local data in an inconsistent state, such as marking a task as saved when it was not, or deleting local data before confirming remote deletion. Graceful handling makes remote operations atomic from the user's perspective: either fully succeeding, with local and remote in sync, or fully failing, with local data unchanged and a clear message, rather than partially succeeding, with local and remote diverged, a divergence that is hard to reconcile and that undermines trust. Finally, error paths must be tested, not just intended. Errors are often untested and buggy, because they are less frequently exercised during development, where success is the focus, while errors, though less frequent in development, are frequent in production, where they must work correctly. Testing with simulated failures, such as a server returning errors, a network timing out, or a rate limit being hit, verifies that the client responds correctly, rather than assuming error handling works because success handling works, an assumption that is false. Tests should cover each error type and assert on the client's response, including its message, its logging, its retry behaviour, and its data safety, such as not corrupting local data when a remote save fails. That testing discipline, applied to API errors, as to other errors, ensures graceful handling is verified, through tests that simulate each error type, rather than hoped for.

Check: Why is a false success, where a failed API operation looks successful, worse than a clear failure?

Section 10: Pagination and Large Result Sets

Pagination is how APIs return large result sets in manageable chunks, called pages, rather than all at once, which would be inefficient, slow, and potentially overwhelming for client, server, and network, especially for datasets with thousands or millions of items, where a single response containing all items would be huge, slow to generate, slow to transmit, and slow to parse, and might exceed size limits or timeouts, causing failures that pagination avoids. A client must handle all of that to retrieve all data, or to retrieve enough data for its task, without assuming a single response contains everything, an assumption that fails for paginated APIs, where the first page is only a subset. A client that processes only the first page will miss most data, producing incomplete results that look complete, because the client does not realise pagination exists, or does not implement it. That silent incompleteness is a common API client bug, where a program that works for small datasets, where all data fits on one page, fails for large datasets, where data spans many pages, because the client was never tested with paginated data, or was tested only with small data that did not trigger pagination. Pagination-aware testing, with mock responses that paginate, closes that gap, by verifying the client follows pages correctly, until all data is retrieved, or until it has enough, rather than stopping after the first page. That verification should be part of every API client's tests, where pagination is relevant, which is for most APIs that return lists, because lists are commonly paginated, even if documentation does not emphasise it. A client that does not handle pagination will inevitably encounter it in real use, when datasets grow beyond one page, a growth that is inevitable for successful APIs, where data accumulates over time. That loop wastes requests, hits rate limits, and never completes, a bug that pagination tests, with mock paginated responses, catch, by verifying the client advances correctly and terminates, after a known number of pages, rather than looping indefinitely. Pagination code, though conceptually simple, is easy to get wrong in detail, such as off-by-one errors in page numbers, or mishandling of cursor tokens, where a token must be passed exactly as received, without modification, to retrieve the next page. Manual URL construction often breaks that exactness, by encoding tokens incorrectly, while parameter-based construction, through a client library, handles it correctly, a reason to use parameter mechanisms for pagination, as for other query parameters, rather than building pagination URLs manually. Finally, for large paginated retrievals, progress logging helps users understand retrieval is proceeding, not stuck, especially for retrievals that take many pages and significant time. That visibility also makes recovery possible, such as resuming from the last successful page, rather than restarting from scratch, which would waste time and requests. Well-designed pagination clients support resumption, by tracking progress and allowing restart from a checkpoint, rather than always starting from page one, a feature valuable for very large retrievals, where restarting from scratch after a failure near the end would be wasteful.

Check: Why must an API client follow pagination instead of assuming one response contains all results?

Section 11: Timeouts and Retries

A timeout is a limit on how long a client will wait for a response before giving up, and setting timeouts is essential, because without them, a request to a slow or unreachable server can hang indefinitely, freezing your program and leaving the user with no feedback and no way to recover except to force-quit. A sensible timeout balances patience with responsiveness: too short, and legitimate slow responses are cut off, causing failures for requests that would have succeeded with a little more time; too long, and users wait far too long for a response that will never come, especially when a server is down, where waiting the full timeout for every request multiplies delays. Different operations deserve different timeouts, with a simple data fetch often given a shorter timeout than a large upload or a complex query, which may legitimately take longer, a distinction that should be based on the operation's nature, not on a single global value applied blindly to everything, though a global default, with per-operation overrides where needed, is a reasonable starting point, provided the default is chosen thoughtfully, not left at a library's default without consideration, which may be far too long, or even infinite, for interactive use. Retries are attempts to repeat a failed request, and they must be used selectively, as discussed for status codes: retry server errors, timeouts, and network failures, where a repeat attempt might succeed, but do not retry client errors unchanged, where the request itself is wrong, because repetition will fail again and waste time and server resources. Retries should also be limited in number, such as a small number of attempts, and spaced with delays, rather than immediate, because immediate retries against a struggling server add load when it is least able to cope, worsening its problems, while a delay gives the server time to recover, and also gives transient network problems time to clear, increasing the chance that a later attempt succeeds, where an immediate attempt would fail again. That delay should grow between attempts, a pattern called exponential backoff, where each wait is longer than the previous one, such as waiting a short time, then a longer time, then longer still, up to a maximum, which spreads retry load over time and avoids a thundering herd, where many clients retry at once, overwhelming a recovering server, a pattern that backoff prevents, by staggering retries, especially when combined with a small random variation, called jitter, which further spreads retry times, avoiding synchronisation where many clients, all waiting the same delay, retry at the same moment, a synchronisation that jitter breaks, by making each client's wait slightly different, a detail that matters for services with many clients, where synchronised retries can cause repeated overload, even with backoff, if all clients use identical delays, a failure that jitter, though it seems minor, prevents, by desynchronising retry timing, a small addition with a large effect at scale. Finally, retries must be safe for the operation being retried.

Check: Why should retries use growing delays, and why is retrying a POST riskier than retrying a GET?

Section 12: Rate Limits and Fair Use

A rate limit sets how many requests a client may make in a given period; a provider might allow a certain number of requests per minute or per hour. The limit protects the service from overload; it also shares capacity fairly among many clients; without a limit, one greedy client could crowd out everyone else. That client might also collect data far faster than the provider intended.

Limits are usually described in the documentation for the service; they may apply per key, per account or per address; they may also differ by endpoint. A read endpoint might allow more requests than a costly search endpoint; you should read those rules before you write your client. Do not assume that a limit that works for one endpoint works for all of them.

A steady pace is better than a burst followed by a long pause; bursts create sudden spikes of load for the provider. They also risk hitting the limit at the worst moment; a steady client spreads its requests evenly over time; it finishes its work without sudden stops. It is also easier to reason about; you can predict when it will finish; you can also plan retries around a known pace.

Many services return headers that describe the current limit; those headers may show how many requests remain; they may also show when the limit resets. Your client should read those values when they are present; it should slow down before the limit is reached; waiting until an error arrives is wasteful. It forces a retry and delays the result.

When a limit error does arrive, respect it; do not retry at once in a tight loop; that loop adds load and may extend the block. Back off for a sensible interval; use the reset time if the service provides one; log the event so you can see the pattern later. Repeated limit errors are a signal; they mean your pace or your design needs to change.

A simple pacing helper can keep a client polite; it records the time of the last request; it waits until enough time has passed before the next request. The example below uses a minimum gap between requests; it is deliberately simple; real clients may use a token bucket or a queue. The principle is the same; space the work out and stay inside the limit.

import time

class Pacer:
    def __init__(self, min_gap_seconds):
        self.min_gap = min_gap_seconds
        self.last_time = 0.0

    def wait(self):
        now = time.monotonic()
        delay = self.min_gap - (now - self.last_time)
        if delay > 0:
            time.sleep(delay)
        self.last_time = time.monotonic()

Fair use goes beyond the numeric limit; it means requesting only the data you need; it means avoiding duplicate requests for the same result. It means caching a result when reuse is allowed; it means stopping when the task is done; a polite client is also a reliable client. It is less likely to be blocked; it also leaves capacity for other users; treat the shared service as a shared resource. Pace your work, respect the signals and keep your requests purposeful.

Check: Why is steady pacing within a rate limit better than bursting requests and then waiting after limit errors?

Section 13: Reading API Documentation

Good documentation is the first place to look when you use a new API; it is written by the people who built the service. It describes the endpoints, the parameters and the expected responses; it also explains the errors you may meet; reading it first saves time. Guessing costs time; copying an example without understanding it costs even more time later.

Start with the overview or getting started page; that page usually explains the purpose of the service; it shows how to authenticate in general terms. It may show a complete example request and response; read that example slowly; note the base address, the path and the method. Note which parts are fixed and which parts you must supply; that distinction matters when you adapt the example.

Next, read the reference for the endpoint you actually need; a reference entry usually lists the method and the path; it lists each parameter and whether it is required. It describes the type of each parameter; it also shows an example response; pay attention to fields that may be missing; a field that is present for one item may be absent for another. Your code must handle that case.

Look for the error section as well; it explains the common status codes for that endpoint; it may explain what a particular error means. It may also explain how to fix it; knowing the errors in advance changes how you write code; you can handle a missing item differently from a bad request. You can handle an authentication problem differently from a server problem; that clarity comes from reading, not from trial and error.

Check the notes on limits, pagination and filtering too; these topics often sit apart from the main reference; they still affect every call you make. A list endpoint may return only a page of results; a search endpoint may need a query parameter in a specific form. A date parameter may need a specific format; missing one of these details leads to confusing results; the documentation usually states them plainly.

Examples in documentation are starting points, not finished programs; they may omit error handling for brevity; they may use placeholder values for keys or identifiers. They may show only the happy path; your job is to turn the example into robust code; add checks for the status code. Add checks for missing fields; add a timeout; test with real responses where you are allowed to do so.

import json
import urllib.request

def fetch_json(url):
    request = urllib.request.Request(url, headers={"Accept": "application/json"})
    with urllib.request.urlopen(request, timeout=10) as response:
        return json.loads(response.read().decode("utf-8"))

Documentation can be incomplete or out of date; that does not make guessing better; it makes careful reading more important; compare the documented response with a real response. Note any difference in your own notes; keep a short record of the endpoints you use; include the parameter names and the fields you rely on. That record helps future you; it also helps anyone else who maintains the client; read first, code second and verify as you go.

Check: Why should you read API documentation before coding, rather than guessing or copying example code?

Section 14: Designing a Simple API Client

A simple API client gathers all the HTTP work in one place; other parts of the program call the client; they do not build requests themselves. They do not parse raw responses themselves; this separation keeps the program easier to understand; it also makes change easier; when the service changes, you change one module, not many scattered calls.

Start by choosing a clear name for the client class; give it the base address once, at construction; store any shared settings there too. Those settings might include a timeout and default headers; they might include a user agent string that identifies your program; keeping them in one place avoids duplication. Duplication is where inconsistencies begin; one call might use a short timeout while another uses none.

Give the client one method per operation your program needs; a method might fetch a single item by identifier; another might search with a query string. Each method should take ordinary Python values as arguments; it should return ordinary Python values, such as a dictionary or a list. Callers should not need to know about HTTP details; they should not need to know about status codes or header names; the client translates between the program and the service.

Error handling belongs inside the client as well; the client should check the status code of each response; it should raise a clear exception for a failed request. That exception should carry useful information; it might include the status code and a short message; it should not include secrets; callers can then decide what to do. They might show a message, retry or skip the item; they make that decision once, in one place.

Keep secrets out of the client code itself; read keys and tokens from configuration or the environment; pass them into the client at construction. Never write a real key into source code; never print a key in a log message; a client that takes its credentials as arguments is easier to test. You can pass a dummy value in tests; you can pass the real value only when the program runs for real.

The example below shows the shape of a small client; it uses only the standard library; it sets a timeout on every request. It checks the status before parsing the body; it raises a clear error when the request fails; you can extend this shape gradually. Add pagination when you need it; add retry logic when you need it; add caching only when you have a reason to cache.

import json
import urllib.request
import urllib.error

class ApiError(Exception):
    pass

class SimpleClient:
    def __init__(self, base_url, timeout=10):
        self.base_url = base_url.rstrip("/")
        self.timeout = timeout

    def get_item(self, item_id):
        url = f"{self.base_url}/items/{item_id}"
        request = urllib.request.Request(url, headers={"Accept": "application/json"})
        try:
            with urllib.request.urlopen(request, timeout=self.timeout) as response:
                return json.loads(response.read().decode("utf-8"))
        except urllib.error.HTTPError as error:
            raise ApiError(f"Request failed with status {error.code}") from error

A central client also improves testing; you can replace it with a fake in tests; you can check that callers handle its errors. You can verify that it builds the right requests; none of that is easy when HTTP calls are scattered; build the client first. Keep it small; let it grow only as your program needs more operations.

Check: Why is a centralised API client better than scattering HTTP calls throughout a program?

Section 15: Testing API Clients with Fakes

Tests for an API client should usually avoid real network calls; real calls are slow; they depend on a service that may be down. They may return different data from one run to the next; they may also hit a rate limit during a test run. Any of these problems can make a test fail for the wrong reason; a test that fails for the wrong reason teaches you to ignore failures. That habit hides real bugs.

Fake responses solve this problem; a fake stands in for the service during a test; it returns a fixed response that you control. Your client code runs exactly as it would in production; only the transport layer is replaced; you can test the happy path with a normal response. You can test error handling with a failure response; you can test missing fields with a partial response; each case runs quickly and repeats reliably.

There are several ways to build a fake; the simplest is a small fake client class; it implements the same methods as the real client. It returns canned data instead of making requests; code that uses the client does not need to change; you pass the fake in place of the real client. This approach works well when your program already uses a central client; it is one more reason to centralise HTTP work.

Another approach replaces the network function used by the client; you can substitute a function that returns a prepared response object; this approach tests more of the real client code. It checks that the client builds the right request; it checks that it parses the response correctly; it also checks that it raises the right error for a bad status. It needs a little more setup; the payoff is stronger coverage of the client itself.

Choose fake data that resembles real data; use realistic field names and realistic values; include an item with a missing optional field. Include an empty list case; include a case where a number arrives as a string, if the service might do that; these cases reveal assumptions in your code. They reveal them in tests, where they are cheap to fix; they do not reveal them in production, where they affect users.

class FakeClient:
    def __init__(self, items):
        self.items = items

    def get_item(self, item_id):
        return self.items[item_id]

def test_missing_item_raises_key_error():
    client = FakeClient({"1": {"name": "Alpha"}})
    try:
        client.get_item("missing")
    except KeyError:
        return True
    return False

Keep a small number of live checks separate from unit tests; a live check might run manually or on a schedule; it confirms that the service still behaves as expected. It is not part of the fast test suite; it may fail for external reasons; that is acceptable when it is labelled as a live check. Unit tests, by contrast, must be fast and stable; fakes give you that stability; use them for most client tests; reserve real calls for deliberate, infrequent verification.

Check: Why should most API client tests use fake responses instead of real API calls?

Section 16: Validating Data from an API

You should validate data from an API even when the service is well documented and trusted; documentation can lag behind the service. A field can be added, renamed or removed without notice; a value can be missing for some items but present for others. A type can differ from what the documentation promises; a server error can return a body that looks like data but is not. Code that assumes perfect data will fail in confusing ways; worse, it may process wrong data silently.

Validation means checking data before you use it; start with the shape of the response. Is it a dictionary when you expect a dictionary? Is it a list when you expect a list? If the shape is wrong, stop and report an error; do not try to guess what the data means; a clear error is better than a wrong result. It tells you exactly where the problem began.

Next, check the fields your program actually needs; confirm that each required field is present; confirm that it has the expected type. A name should be a string; a count should be a number; an identifier should be present and usable; if a required field is missing, handle that case explicitly. You might skip the item and log a warning; you might raise an error; either choice is better than crashing later in unrelated code.

Optional fields need a different approach; give them a sensible default when they are absent; do not assume that an optional field will always appear. A short description might be missing; an image link might be empty; your program should still work without them; defaults keep the program running. They also make the missing data visible in a controlled way.

Be careful with values that look valid but are not useful; an empty string is not a useful name; a negative count may not make sense. A date string in an unexpected format may fail to parse; check ranges and formats where they matter; reject data that cannot be trusted. Log what you rejected and why; that log helps you spot a change in the service; it also helps you explain a missing item to a user.

def validate_item(data):
    if not isinstance(data, dict):
        raise ValueError("Item must be a dictionary")
    name = data.get("name")
    if not isinstance(name, str) or not name.strip():
        raise ValueError("Item needs a non-empty name")
    count = data.get("count", 0)
    if not isinstance(count, (int, float)) or count < 0:
        raise ValueError("Item count must be zero or more")
    return {"name": name.strip(), "count": count}

Validation also protects later code; once data passes validation, the rest of the program can trust its shape; it does not need to re-check every field. That trust is earned at the boundary; it is the boundary between the outside world and your program; keep that boundary strict. Keep the inside of your program simple; validate early, fail clearly and never silently accept data you cannot understand.

Check: Why should you validate API response data even when the API is well documented and trusted?

Section 17: Transforming API Data for Your Program

API data often arrives in a shape designed for general use; that shape may not suit your program; field names may be long or unclear. Nested structures may bury the values you need; extra fields may clutter your code; transforming the data at the boundary solves these problems. You convert the external shape into an internal model; your program then works with a shape it understands.

An internal model is usually a simple class or a dictionary with chosen keys; it holds only the fields your program needs. It uses names that make sense in your program; it stores values in types that are easy to work with; for example, a date string might become a date object. A nested author object might become a plain author name; these conversions happen once, at the boundary; they do not happen again in every part of the program.

This separation pays off when the API changes; suppose the service renames a field; you update the transformation in one place; the rest of your program keeps working. Suppose the service nests a value one level deeper; you update the same place; without this boundary, you would search the whole program for every use of that field. You might miss one; that missed use becomes a bug.

Transformation also clarifies your program; code that reads an internal model is easy to follow; it does not wade through nested dictionaries. It does not guess whether a key exists; it uses attributes or keys with known meanings; new readers can understand the model quickly. They can see exactly what data the program cares about; they do not need to study the full API response.

Keep the transformation honest and simple; copy the values you need; convert types where conversion is safe; apply defaults for missing optional fields. Reject data that fails validation before transforming it; do not invent values that the service did not provide; do not silently drop a field that matters. If a value cannot be converted, raise an error; a clear failure at the boundary is easier to debug than a wrong value downstream.

class Article:
    def __init__(self, title, author, word_count):
        self.title = title
        self.author = author
        self.word_count = word_count

def to_article(data):
    return Article(
        title=data["title"],
        author=data.get("author", {}).get("name", "Unknown"),
        word_count=len(data.get("body", "").split()),
    )

The example shows a small transformation; it picks three fields from a larger response; it supplies a default author when the author is missing. It computes a word count from the body text; the rest of the program can now use Article objects; it never touches the raw response again. That boundary keeps API concerns apart from application concerns; build the model for your program, not for the service; transform once, at the edge, and let the rest of the code stay clean.

Check: Why transform API data into internal models at the boundary, instead of using API data directly throughout your program?

Section 18: Combining API Data with Local Data

Many programs combine API data with data stored locally; local data might include user preferences; it might include notes, favourites or records created in the program. It might include cached results from an earlier session; combining these sources is useful; it is also a source of subtle bugs. The two sources can disagree; they can drift apart over time; your program needs a plan for that disagreement.

The first question is authority. Which source wins when they conflict? There is no universal answer; it depends on the kind of data; for a user preference, the local value may win; the user set it deliberately. For a published fact, the API value may win; the service is the source of truth; for a record the user edited locally, the local edit may win until it is sent back. Decide this rule per field, not per program; different fields may need different rules.

The second question is identity. How do you know that a local record and an API record describe the same thing? You need a stable identifier; that identifier might be an id supplied by the service; it might be a key you generated locally. Match on that identifier, not on a name or a title; names can change; titles can repeat; a stable id gives you a reliable join. Without it, merging becomes guesswork.

The third question is timing; data changes at different rates; local data may be edited while the program is offline; API data may change on the server at any time. A merge that ignores timing can lose an update; one simple approach is to record when each value was last changed; the newer value wins, when recency is the right rule. Another approach keeps both values and asks the user; that approach suits important conflicts; it does not suit every small field.

Merging also needs a clear result; a merged record should be complete and consistent; it should not mix half of one source with half of the other by accident. Build the merged record in one function; that function applies your authority rules; it applies your timing rules; it returns a single record that the rest of the program can use. Keeping the logic in one place makes it testable; you can test each conflict case with small examples.

def merge_item(local, remote):
    merged = dict(remote)
    merged.update(local.get("overrides", {}))
    merged["id"] = remote["id"]
    return merged

The example is deliberately small; remote data provides the base record; local overrides replace specific fields; the remote id is always kept. A real program would add more rules; it might check timestamps; it might flag conflicts for review; it might queue local changes to send later. Whatever the rules are, write them down; state which source wins and why; a program that combines sources without a rule will eventually show inconsistent data. A program with a clear rule will behave predictably, even when the sources disagree.

Check: Why must a program that combines local and API data define which source wins when they conflict?

Section 19: Caching API Responses

Caching means storing an API response locally so it can be reused; the next request for the same data reads the local copy. No network request is needed; this makes the program faster; it reduces bandwidth; it reduces load on the service; it also lets the program work, at least partly, when the network is unavailable. These benefits are real; they come with a trade-off that must be managed.

The trade-off is staleness; cached data ages; the remote data may change after you store your copy; your program may then show an old value. That old value may mislead the user; a price, a status or a count can change quickly; showing yesterday's value as if it were current is a bug. It is a quiet bug, because the program appears to work; the user has no reason to suspect the data is old.

Every cache therefore needs an expiry or invalidation rule; expiry means a cached entry is used only for a limited time; after that time, it is refreshed. A short expiry suits fast-changing data; a long expiry suits data that rarely changes; invalidation means removing or refreshing an entry when you know it changed. An update action might invalidate the related entry; a signal from the service might do the same; either rule prevents a cache from serving ancient data forever.

Choose the rule based on the data; ask how quickly the data changes; ask how harmful a stale value would be; a list of categories might safely live for a day. A live status should not live for a day; a user profile might sit between those extremes; write the rule down next to the cache. Future maintainers should see why the expiry is what it is; they should not have to guess.

Users should also be able to see the age of cached data; show a simple label when data may be stale; the label might say when the data was last refreshed. It might say that the data is cached; this honesty lets users judge the result; they can refresh manually when freshness matters. They can also understand why a value differs from another source; hiding the age of data removes that judgement.

import time

class Cache:
    def __init__(self, max_age_seconds):
        self.max_age = max_age_seconds
        self.store = {}

    def get(self, key):
        entry = self.store.get(key)
        if entry is None:
            return None
        value, saved_at = entry
        if time.monotonic() - saved_at > self.max_age:
            return None
        return value

    def put(self, key, value):
        self.store[key] = (value, time.monotonic())

A cache also needs limits; an unbounded cache grows without end; set a maximum size; remove the oldest entries when the limit is reached. Handle the case where the cache is empty or expired; fall back to a fresh request in that case; if the fresh request fails, you may show the stale value with a clear warning. That fallback is a choice, not an accident; make it deliberately; cache with a rule, show the age and keep the size bounded.

Check: Why does every cache need an expiry or invalidation rule, and why should users be able to see data age?

Section 20: Security and Privacy with APIs

Security and privacy matter in every API interaction; APIs often carry personal data; they may carry private records or financial details; mishandling that data can cause serious harm. Harm can come from insecure transport; it can come from exposed credentials; it can come from excessive logging; it can come from collecting more data than you need. Secure habits prevent most of these problems; they are not optional extras.

Use HTTPS for all API communication; HTTPS encrypts data in transit; it prevents others on the network from reading the data; it also prevents them from modifying it unnoticed. This protection matters even for data that does not seem sensitive; a request can reveal what a user is interested in; a response can reveal patterns of behaviour. Metadata alone can be revealing; do not send API traffic over plain HTTP; if a service offers only plain HTTP, treat that as a serious warning.

Protect credentials with the same care; a key or token is a secret; keep it out of source code; keep it out of screenshots and shared logs. Read it from configuration or the environment at run time; pass it only where it is needed; use the narrowest permission the service allows. A key that can only read public data is less dangerous if leaked than a key that can change data; rotate a key promptly if you suspect it leaked. Do not wait to confirm the damage.

Be careful with logging; logs are useful for debugging; they are dangerous when they capture secrets or personal data; never log a full request that includes a credential. Never log a full response that includes personal details; log the status code and a short summary instead; review what your program logs before you share those logs. A log file can travel further than you expect; treat it as a record that others may read.

Collect only the data you need; this is called minimisation; if your program needs a name, request a name; do not request a full profile when a name will do. If your program needs a count, request a count; do not download a full dataset to count it locally; less data means less risk. It also means less work to protect; delete data when you no longer need it; do not keep copies indefinitely out of habit.

import os
import urllib.request

def authorised_request(url):
    token = os.environ.get("EXAMPLE_API_TOKEN", "")
    request = urllib.request.Request(url)
    if token:
        request.add_header("Authorization", f"Bearer {token}")
    return urllib.request.urlopen(request, timeout=10)

The example reads a token from the environment; it never writes the token into code; it never prints the token; it adds the token only to the request header. That pattern is a good default; combine it with HTTPS, careful logging and minimal collection; review your client for each of these habits. Security is not a single feature; it is a set of small decisions, made consistently, every time your program talks to a service.

Check: Why should API communication use HTTPS even for data that does not seem sensitive?

Section 21: Webhooks in Concept

A webhook is a reverse API call. Instead of your program asking a service for updates, the service calls your program when something happens. For example, a service might notify your program when a new record is created. Your program provides an address that can receive such notifications. The service sends an HTTP request, usually a POST, to that address when the event occurs.

Webhooks are useful because they avoid polling. Polling means asking repeatedly whether anything has changed. That wastes requests when nothing has changed. It also delays news, because changes are only noticed at the next poll. A webhook delivers news promptly, with only one request per event. That efficiency matters for programs that track many events. It also reduces load on the service, which is a courtesy as well as a practical benefit.

Receiving webhooks brings responsibilities. Your receiving endpoint must be reliable and fast. If it is slow, the sender may time out and retry, causing duplicate notifications. Your code should therefore handle duplicates gracefully, by checking whether an event was already processed. It should also verify that notifications really come from the claimed sender, using the verification method the service documents, rather than trusting any request that arrives. Unverified webhook endpoints can be abused by anyone who discovers the address.

Finally, webhooks are not always the right choice. They require your program to be reachable from the internet, which a simple local script is not. They also require handling retries, duplicates, and verification, which adds complexity. For a small learning program, polling on a schedule may be simpler and sufficient. Choose webhooks when prompt delivery matters and when you can run a reachable receiver. Choose polling when simplicity matters more, or when your program runs only occasionally. Understanding both patterns lets you pick the right one for each task, rather than forcing one pattern onto every problem.

In practice, a webhook receiver should respond quickly and do heavy work afterwards, not during the request itself. If the receiver takes too long, the sender may assume delivery failed and send the event again. A good pattern is to accept the notification, store it safely, reply promptly, and then process the stored event in a separate step. That separation keeps delivery reliable even when processing is slow or fails temporarily. It also makes retries safe, because reprocessing a stored event can be checked against events already handled, rather than repeating side effects blindly. Designing receivers this way turns webhooks from a fragile shortcut into a dependable integration method.

Testing a webhook receiver is also straightforward once you separate receiving from processing. You can send it a sample notification yourself, using a small script, and check that it stores the event and replies promptly. You can also send the same notification twice, to confirm that the second copy is recognised as a duplicate and does not repeat side effects. Those two tests cover the most common webhook failures, and they can be run without waiting for a real event from the service.

Check: What is a webhook, and why must a webhook receiver handle duplicate notifications?

Section 22: API Versioning

APIs change over time, and versioning is how providers manage those changes without breaking every existing client at once. A version is a label, often in the URL path, such as /v1/ or /v2/, that identifies which set of endpoints and data shapes a client is using. When a provider makes a breaking change, such as renaming a field or removing an endpoint, it can release a new version, while keeping the old version working for a transition period. Existing clients continue to work on the old version, while new clients use the new version. That transition period gives developers time to update, test, and deploy, rather than facing a sudden breakage.

Not all changes are breaking. Adding a new field, or a new endpoint, usually does not break existing clients, because they can ignore what they do not use. Removing a field, renaming a field, or changing a field's type, by contrast, can break clients that depend on the old shape. Providers therefore distinguish breaking changes, which require a new version, from non-breaking additions, which can be made within a version. As a client developer, you should read versioning and deprecation notices carefully, because they tell you when a change is coming, what will break, and how long you have to adapt. Ignoring those notices risks a sudden failure when an old version is finally retired.

Your own code should make its API version explicit, rather than relying on a default that may change. If a version is in the URL, include it in your client's base address, in one place, so that upgrading means changing one value, not many scattered URLs. If a version is in a header, set it in your client's shared configuration, for the same reason. Centralising the version, like centralising other client settings, makes upgrades manageable and testable. You can then test the new version in isolation, compare its responses with the old version, and switch over deliberately, rather than discovering a version change through failures in production.

Finally, versioning applies to your own APIs too, when you later design services for others. The same principles hold: do not break existing consumers without warning, provide a transition period, and document changes clearly. Learning versioning as a consumer teaches you to be a considerate provider, because you have experienced both sides: the frustration of a sudden breaking change, and the relief of a well-managed transition. That empathy, combined with the practical habits of explicit versions, centralised configuration, and attention to deprecation notices, makes versioning a core professional skill, not just a label in a URL, but a commitment to stability and communication between provider and consumer.

A practical tip is to record which version each client uses, and when each version is scheduled for retirement. That record makes upgrades easy to plan, because you can see at a glance which programs need attention before a deadline. Without such a record, version upgrades are discovered by accident, often too late, when an old version stops working and several programs fail at once.

Check: Why do API providers use versions, and why should your client make its version explicit in one place?

Section 23: Handling API Changes and Deprecations

Deprecation is a warning that something will be removed or changed in the future, but still works for now. Providers deprecate endpoints, fields, or whole versions to give clients time to migrate, rather than breaking them immediately. A deprecation notice usually includes what is deprecated, what to use instead, and when the old thing will stop working. Reading those notices, and acting on them early, is far better than waiting until the deadline, when migration becomes urgent, rushed, and error-prone. Early migration lets you test the replacement at leisure, compare results, and fix problems before the old version disappears.

When you learn that an endpoint or field you use is deprecated, plan the migration as a small project. First, find every place in your code that uses the deprecated thing, which is easier if your client is centralised, as discussed earlier, because uses are in one place, not scattered. Second, read the replacement's documentation, noting differences in parameters, data shapes, and behaviour. Third, implement the replacement alongside the old code, if possible, and test it with fake responses and with real responses in a safe environment, comparing outputs. Fourth, switch over, monitor for errors, and remove the old code only after the new code has proven reliable. That staged approach reduces risk, compared with a sudden, wholesale replacement, which, if it contains a bug, affects everything at once.

Deprecation also affects data you have stored. If a field you stored from an API is deprecated and will change shape, your stored data may need migration too, or your code may need to handle both old and new shapes during a transition. Planning for stored data, not just for live requests, is part of a complete migration, and is often overlooked, leading to failures when old stored data is read by new code that expects the new shape. Handling both shapes during transition, with clear transformation logic, as discussed for data transformation, makes migration smoother, because old data remains readable while new data arrives in the new shape, and a later cleanup can convert old data, once the transition is complete, rather than forcing an immediate, risky conversion of all stored data at once.

Finally, keep a habit of checking for deprecation notices regularly, not only when something breaks. Providers announce deprecations through documentation, changelogs, response headers, or email notices, depending on the service. Checking those channels periodically, as part of routine maintenance, catches deprecations early, when there is time to migrate calmly. That routine, like checking for library updates or reviewing logs, is part of operating a program that depends on external services, where change is constant and must be managed, not ignored. A program that depends on an API is never truly finished, because the API will evolve, and the program must evolve with it, through attentive, staged migrations, that preserve reliability, rather than through panicked fixes, after a deadline has passed.

Check: What staged approach should you follow when an endpoint you use is deprecated?

Section 24: A Simple Public Data Client

Imagine building a small program that fetches public data, such as weather summaries or public dataset listings, and displays a simple report. The design starts not with code, but with questions: which API provides the data, what endpoints are needed, what parameters control the request, what data shape is returned, and what the program will do with that data. It also clarifies scope: a small client should do one thing well, such as fetching and displaying a report, rather than trying to cover every endpoint the API offers, which would be a large, unfocused program, hard to finish and hard to maintain. Next, design the client structure, following the centralised pattern from earlier. Create one client component that holds the base address, the version, timeouts, and error handling, and exposes methods for each operation the program needs, such as fetching the current report, with parameters for location or category, as appropriate. Those methods should validate inputs, construct requests through the parameter mechanism, check status, validate the response shape, and return internal models, transformed from API data, rather than raw API data, so that the rest of the program works with stable, program-specific shapes. That structure, though it may seem like extra work for a small program, makes the program testable, because each method can be tested with fake responses, and maintainable, because API changes affect only the client, not the display logic. Then, design the display and storage parts separately from the client. Display logic should take internal models and present them clearly, without knowing about HTTP, endpoints, or API data shapes. If the program saves data, such as caching a report to a file, that storage should also work with internal models, in a format like JSON or CSV, as taught in the previous chapter, so that saved data remains readable even if the API changes. That separation, client, display, and storage, each with its own responsibility, mirrors good program design generally, where each part does one job well, and where changes in one part do not ripple through the others, a design that makes even a small program easier to understand, to test, and to extend, if it later grows, for example by adding more endpoints, more display options, or more storage, all of which can be added without rewriting the whole program, because each part is independent. Finally, test the whole program in stages. Test the client with fakes, covering success, errors, pagination, and invalid data. Test display with sample internal models, without needing the API. Test the integrated program with a real API call, in a safe, low-volume way, verifying that the report matches the API's data, and that errors are handled gracefully, with clear messages, not crashes. That staged testing, from isolated parts to integrated whole, catches problems early, when they are cheap to fix, and gives confidence that the program works, not just in theory, but with real data, in real conditions. That confidence, built through design, separation, and staged testing, is the reward for the structured approach, which, though it takes a little more planning than a quick script, produces a program that is reliable, understandable, and ready to grow, rather than a fragile script that works once, and breaks on the next API change.

Check: Why should a public data client separate API fetching, display, and storage into different parts?

Section 25: Public Datasets and Responsible Use

Public datasets, available through APIs, are a rich resource for learning, for projects, and for building useful programs, covering topics from weather to geography to public statistics. Using them responsibly means respecting the provider's terms, rate limits, and attribution requirements, because public does not mean unlimited or unconditional. Terms may require attribution, may restrict commercial use, or may limit how much data can be downloaded, and those terms, though they may seem like legal detail, are part of the API's contract, and should be read and followed, as with documentation generally, because violating them can lead to blocked access, or to legal problems, depending on the terms and the use, consequences that responsible use avoids, by reading terms first, and by designing programs that comply, such as by including required attribution in displays, by staying within rate limits, and by not redistributing data in ways the terms forbid, all of which are straightforward, once terms are known, but which, if ignored, can cause problems, that, though they may seem unlikely, are real, for programs that become popular, or that are used commercially, where terms matter, and where compliance, from the start, prevents later difficulties, when a program must be changed, or taken down, due to a terms violation, that could have been avoided, by reading terms, before building, a reading that, like documentation reading, saves time, and prevents problems, in the long run. Responsible use also means being honest about data. Public data may contain errors, gaps, or biases, depending on how it was collected, and a program that presents data should not present it as more accurate, more complete, or more authoritative than it is. If data is missing for some items, the program should show that gap, rather than hiding it, or inventing values to fill it, because invented values mislead users, who may act on false data, believing it is real. If data is estimated, or sampled, the program should say so, where it matters, so users can judge its reliability, for their purpose. That honesty, about data limitations, is part of responsible data use, alongside legal compliance, and it builds trust, because users who understand a program's data limitations can use it wisely, while users who are misled about limitations may use it unwisely, with consequences, that, though they may be minor, for a learning project, could be serious, for a program that informs decisions, a seriousness that honest presentation, prevents, by making limitations visible, rather than hidden, a visibility that, like other transparency, in this book, is presented, as a professional duty, not just a courtesy, for programs that present data, to others, where honesty, about sources, limitations, and uncertainty, is essential, for trustworthy programs, that users can rely on, without being misled, by false precision, or false completeness, that, though it may make a program look better, in the short term, undermines trust, in the long term, when limitations, inevitably, become apparent, through errors, or through comparison, with other sources, a comparison that, honest.

Check: Why does responsible use of a public dataset API include reading its terms and being honest about data gaps?

Section 26: Saving API Data to Files

Saving API data to files lets a program keep results for later use, for offline work, or for analysis, without re-requesting the same data repeatedly, which saves time, bandwidth, and rate limit budget, and which, as discussed for caching, allows work to continue, when the API is unavailable, though saved data, like cached data, can become stale, and should be treated, accordingly, with timestamps, and with awareness, that it reflects, the API's state, at the time of saving, not necessarily, its current state, a awareness that, timestamping, and clear labelling, provide, by recording, when data was fetched, from which endpoint, and with which parameters, so that later users, including yourself, understand, what saved data represents, and whether it is, still fresh enough, for their purpose, a understanding that, unsaved metadata, lacks, where saved data, without context, may be mistaken, for current data, or for data, from a different query, a mistake that, metadata, prevents, by documenting, the data's origin, and age, alongside the data itself, a documentation that, though it adds, a small amount, of storage, pays off, in clarity, and in preventing, misuse of stale, or misidentified data, a prevention that, like other metadata, in this book, is presented, as good practice, for saved data, generally, not just for API data, where origin and age, matter, for interpretation, and where, without them, saved data, loses meaning, over time, as its context, is forgotten, a forgetting that metadata, prevents, by preserving context, with data, for future use. The format for saving should match the data and its intended use, as taught in the previous chapter. JSON is suitable for structured, nested API data, because it preserves structure, types, and nesting, in a readable, widely supported format, that can be reloaded, and processed, later, without loss, of structure, though it may be verbose, for large datasets, where a more compact format, might be preferable, a trade-off, that should be considered, per dataset, based on size, and on, how the data, will be used, later, whether for display, for analysis, or for re-import, into another program, all of which, JSON supports, well, for structured data, while CSV, is suitable, for flat, tabular data, such as lists of items, with consistent fields, where CSV's simplicity, and compatibility, with spreadsheets, and analysis tools, make it, a good choice, for data, that will be, analysed, or shared, with non-programmers, who can open CSV, in familiar tools, a sharing that JSON, though readable, to programmers, may be, less accessible, to others, a accessibility that, format choice, should consider, based on audience, and on use, not just on, programmer convenience, a consideration that, like other design choices, in this book, balances, technical suitability, with human usability, for the data's, intended audience, and purpose, a balance that, thoughtful format choice, achieves, by matching format, to data shape, and to use, rather than defaulting, to one format, for everything, a default that, though simpler, may be, unsuitable, for some data, or for some audiences, a unsuitability that, thoughtful choice,.

Check: Why should saved API data include metadata about when and how it was fetched?

Section 27: Scheduled Refreshes

A scheduled refresh re-requests API data at regular intervals; it keeps a local copy up to date without manual action; a dashboard might refresh every hour. A small tool might refresh once a day; the schedule should match how quickly the data changes; it should also match how important freshness is. Refreshing too often wastes resources; refreshing too rarely leaves stale data on display.

Choosing an interval is a design decision; start with the needs of the user. How old can the data be before it misleads? A weather summary tolerates an hour of age; a live status may not tolerate an hour; next, consider the load on the service. Frequent refreshes from many clients add up; respect the rate limit; respect the terms of use; a schedule that ignores those limits will eventually fail. It may also get the client blocked.

Unattended refreshes need special care; no user is watching when they run; a failure may go unnoticed for days; a silent failure is worse than a visible one. The local copy keeps ageing; users keep seeing old data; they assume it is current; that assumption is the danger; logging and alerting prevent it. Logging records what happened on each run; alerting tells a human when something needs attention.

Log the essential facts of each run; record when the run started and finished; record whether it succeeded; record how many items were fetched. Record the error message when it failed; keep the log readable and bounded; a log that grows without limit becomes useless; a log that captures secrets becomes dangerous. Log summaries, not full responses and never credentials.

Alerting turns a log into action; an alert might be an email or a message to the maintainer; it should fire when failures repeat. A single failure may be a temporary outage; repeated failures suggest a deeper problem; the service may have changed; the credentials may have expired. The schedule may be hitting a limit; an alert brings that problem to a human promptly; retry logic alone does not do this. Retries may keep failing quietly; they may also add load during an outage.

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("refresh")

def refresh_once(fetch):
    try:
        items = fetch()
    except Exception as error:
        logger.error("Refresh failed: %s", error)
        return False
    logger.info("Refresh succeeded with %d items", len(items))
    return True

Test the scheduled job before relying on it; run it manually once; check the log output; confirm that a failure produces a clear log entry. Confirm that repeated failures would trigger an alert; confirm that a successful run updates the data visibly; keep the job simple; keep its interval sensible. A scheduled refresh is a promise to keep data fresh; logging and alerting are how you keep that promise when nobody is watching.

Check: Why do unattended scheduled API refreshes need logging and alerting, not just retry logic?

Section 28: Debugging API Calls

Debugging an API call means finding why a request did not produce the expected result; the best approach is systematic; check each part of the request in turn. Check each part of the response in turn; guessing wastes time; changing code at random may appear to fix the problem; it often introduces a new problem. A systematic check finds the real cause; it also finds it faster in total, because it avoids false fixes.

Start by reproducing the problem with a minimal request; use the simplest request that still fails; remove extra parameters; remove extra headers. If the minimal request succeeds, add parts back one at a time; the part that breaks it is the suspect; if the minimal request fails, you have a small case to study. A small case is easier to share and easier to reason about.

Next, examine the request itself. Is the method correct? Is the path correct? Is the base address correct? A small typo in a path produces a confusing error; check the query parameters; check their names and their values; check that values are encoded correctly. Spaces and special characters need encoding; check the body when the method sends one; check that the content type matches the body. A body sent as plain text will not parse as JSON.

Then examine the response; start with the status code; it tells you the broad category of the result; a client error points to your request. A server error points to the service; read the response body for detail; it may contain a message that explains the failure. It may name the field that was invalid; do not ignore that message; it is often the fastest clue; also check the response headers. They may show a rate limit or a content type problem.

Compare your request with the documentation; re-read the endpoint reference; confirm the required parameters; confirm the expected types; confirm the authentication method in general terms. Many bugs come from a detail that was skimmed; a required parameter may be missing; a parameter may be in the wrong place. It may be in the query string when it belongs in the body; the documentation usually answers these questions.

import json
import urllib.request
import urllib.error

def debug_get(url):
    request = urllib.request.Request(url, headers={"Accept": "application/json"})
    try:
        with urllib.request.urlopen(request, timeout=10) as response:
            print("Status:", response.status)
            print("Body:", response.read().decode("utf-8")[:500])
    except urllib.error.HTTPError as error:
        print("Status:", error.code)
        print("Body:", error.read().decode("utf-8", errors="replace")[:500])

Keep notes as you debug; write down what you checked and what you found; that record prevents repeated checks; it also helps when you ask for help. You can show the exact request and the exact response; you can show which parts you already ruled out; systematic debugging is not slower. It is careful work that respects your time; check the request, check the response, compare with the documentation and change one thing at a time.

Check: Why is systematic checking of each request part better than guessing when debugging an API call?

Section 29: Common API Pitfalls

Every programmer who works with APIs eventually meets the same set of failures, and learning them in advance is far better than discovering each one through a broken program of your own. The first group of pitfalls concerns assumptions about responses. A program that assumes a field will always be present, that a list will never be empty, or that a value will always have the same type, will work against the examples in the documentation and fail against real data, where fields are sometimes absent, lists are sometimes empty, and types are sometimes surprising. Validating what arrives, and handling absence gracefully, is not pessimism; it is the recognition that an API's data is produced by another system, with its own bugs, migrations, and edge cases, and that your program is the place where those surprises become visible.

A second group concerns behaviour over time. An API that responds quickly during development may be slow under load, may enforce limits you have not yet reached, or may change in ways that break a program that depended on an undocumented detail. Programs that make requests without timeouts, that retry failures immediately and repeatedly, or that depend on the order of results when no order is promised, are all waiting for a bad day. The same applies to programs that ignore status codes and try to parse every response as success: an error page is not data, and treating it as data produces confusing failures far from the real cause. Checking the status first, respecting limits, and backing off before retrying are habits that cost little during development and save a great deal when conditions change.

A third group concerns the program's own assumptions about identity and repetition. Sending the same request twice, whether because of a retry or a double click, can create a duplicate record if the API does not guard against it, and a program that assumes a request succeeded because it was sent, rather than because a success response arrived, can lose work silently. Keeping track of what was actually acknowledged, designing operations that can safely be repeated, and logging enough detail to reconstruct what happened are all part of avoiding this group of pitfalls. None of these lessons is difficult on its own, but each one is painful to learn from a production failure, when real users are affected and the cause must be found under pressure.

def handle(response_status, data):
    if response_status != 200:
        return "error"
    items = data.get("items") or []
    return len(items)
print(handle(200, {"items": [1, 2, 3]}))

Learn the common pitfalls before they find you: validate what arrives, check the status before parsing, set timeouts, respect limits, and never assume a request succeeded merely because it was sent. A program built with those cautions will still meet new failures, but it will meet them as manageable errors with clear messages, rather than as silent corruption or mysterious crashes. That is the practical payoff of studying pitfalls in advance: not the avoidance of all failure, which is impossible, but the avoidance of the failures that were already well known.

Check: Why is learning common API pitfalls in advance better than learning only through your own failures?

Section 30: API Etiquette and Terms of Use

Using an API politely is often described as good manners, but following its terms of use is a practical necessity, because the terms describe the conditions under which the service will keep working for you. A provider may limit how many requests you may make, forbid certain kinds of use, require attribution, or prohibit storing particular data beyond a stated period. Those rules are not decoration; they are enforced, often automatically, by rate limiting, by blocking, or by suspending access. A program that ignores them may work for a while and then stop abruptly, at the moment its user needs it most, with no warning beyond the terms it never read. Reading the terms before building is therefore part of designing a reliable program, not an administrative chore to be skipped.

Etiquette extends beyond the written terms into the everyday behaviour of a well-behaved client. Making requests only as often as needed, caching responses instead of fetching the same data repeatedly, and avoiding bursts of requests that hammer a shared service all reduce load on the provider and reduce the chance that your program will be throttled. Sending a clear identifier for your client, so the provider can contact you if something goes wrong, is similarly practical: an anonymous flood of requests looks like abuse, while an identified client with a reasonable pattern looks like a legitimate user. These behaviours cost almost nothing to implement, and they protect your program's access in the same way that considerate driving protects your licence: not by guaranteeing good outcomes, but by avoiding the self-inflicted ones.

Terms of use also carry legal and ethical weight that a programmer cannot safely ignore. Data obtained through an API may come with restrictions on redistribution, on combining it with other data, or on using it for particular purposes, and those restrictions may apply even when the data itself is publicly visible. A program that republishes restricted data, or that uses it in a prohibited way, can expose its author to account closure, legal claims, or reputational harm, regardless of whether the program worked technically. Checking what is permitted, and designing the program to stay within it, is therefore a design constraint like any other, to be considered early, when it is still cheap to change direction.

CACHE = {}
def fetch(key, loader):
    if key not in CACHE:
        CACHE[key] = loader(key)
    return CACHE[key]
print(fetch("summary", lambda k: "cached data"))

Treat an API's terms of use as part of its interface, to be read before coding and respected in the program's behaviour. Cache where caching is allowed, request only what you need, identify your client, and never redistribute or reuse data in ways the terms forbid. A program that follows those rules is not merely polite; it is durable, because its access depends on the provider's continued goodwill, and goodwill is maintained by behaviour, not by intentions.

Check: Why is following an API's terms of use a practical necessity, not just politeness?

Section 31: Error Messages for API Users

When an API call fails, the message your program shows matters. A vague message, such as "Error", tells the user nothing useful. A good message says what happened, in plain language. It also says what the user can do next, where possible. That guidance turns a failure into a solvable problem, rather than a dead end.

Different failures need different messages. A missing resource should say the item was not found, and suggest checking the name or identifier. An authentication failure should say sign-in failed, and suggest checking credentials, without revealing technical details. A server failure should say the service is temporarily unavailable, and suggest trying again later. Each message matches the cause, so the user is not misled into fixing the wrong thing.

Technical details still belong somewhere, but not in the user message. Log the status code, the endpoint, and the error body for developers. Keep those details out of the user interface, where they confuse rather than help. That separation, friendly message for users and detailed log for developers, is a habit worth keeping in every program. It protects users from jargon, and gives developers what they need to diagnose problems.

Finally, test your error messages. Trigger each failure type with a fake response, and read the message as a user would. Ask whether it is clear, honest, and actionable. If it is not, rewrite it before release. Error messages are part of the user experience, not an afterthought. Programs are often judged by how they behave when things go wrong, because that is when users need the most help.

Consider a concrete example. A user tries to save a record, but the network fails. A poor message says only that an error occurred. The user does not know whether the record was saved, whether to retry, or whether to check their connection. A good message says the save could not be confirmed, that the record may not be saved, and that the user should check their connection and try again, or check whether the record appears in their list first. That message respects the user's time and data. It also prevents duplicate saves, because the user checks before retrying, rather than retrying blindly.

A second example involves permissions. A user tries to open a shared item, but access was removed. A poor message says access denied, with no explanation. The user may think the program is broken. A good message says the item is no longer shared with them, and suggests contacting the item's owner if they believe that is a mistake. That message is honest, specific, and actionable. It does not reveal who removed access, or why, because those details are private. It gives the user a next step, which is what a helpful message should always do, even when the underlying problem cannot be fixed by the user alone.

Check: Why should user-facing API error messages differ from developer logs?

Section 32: Idempotency

Idempotency means that repeating an operation has the same effect as doing it once. A GET request is naturally idempotent, because reading data does not change it. Repeating a GET gives the same result, with no side effects. That safety makes GET requests easy to retry, as discussed earlier. Creation requests, by contrast, are not naturally idempotent, because repeating a POST can create duplicates.

Some APIs offer an idempotency mechanism to solve that problem. The client sends a unique identifier with each creation request. The server stores that identifier with the created item. If the same identifier arrives again, the server recognises it as a retry, and returns the original result, instead of creating a second item. That mechanism makes retries safe, even when the first response was lost, because the server can distinguish a retry from a genuinely new request. It depends on the client generating a unique identifier per logical operation, not per attempt, so that all attempts of the same operation share the identifier.

Where an API does not offer idempotency, the client must be more careful. Before retrying a creation, it can check whether the item already exists, for example by listing recent items and looking for a match. That check is not perfect, because matching can be ambiguous, but it reduces duplicate risk. Alternatively, the client can ask the user to confirm before retrying, explaining that the first attempt may have succeeded, even though its response was lost. That honesty lets the user decide, rather than risking a silent duplicate.

Understanding idempotency helps you design safer programs in general. When you design your own operations, ask whether repeating them is safe, and if not, how to make them safe. Idempotent operations are easier to retry, easier to test, and easier to reason about, because their effect does not depend on how many times they run. That property, though it may seem abstract, has practical benefits for reliability, especially in networked programs, where requests can be lost, duplicated, or delayed, and where safety under repetition is a real requirement, not a theoretical nicety.

Idempotency also matters beyond APIs, in everyday program design. A file-saving operation that can be run twice, with the same result, is safer than one that appends a duplicate entry each time it runs. A task-completion operation that marks a task done, whether it was already done or not, is safer than one that errors on the second run. Designing operations to be repeatable, where possible, makes programs more robust, because retries, restarts, and duplicate triggers, which are common in real systems, do not cause harm. That robustness is worth designing for deliberately, rather than discovering its absence through duplicate data, after a retry, in production.

Check: How does an idempotency identifier make retrying a creation request safe?

Section 33: Formats Beyond JSON

JSON is the most common API data format, but it is not the only one. Some APIs return XML, an older markup format, with tags that describe data structure. Others return plain text, or comma-separated values, for simple data. Still others return binary data, such as images or files, which must be handled differently from text. Knowing that formats vary prevents a common mistake: assuming every response is JSON, and failing when it is not. Checking the content type header, as discussed earlier, tells you which format to expect, before you try to parse the body.

Each format has trade-offs. JSON is compact, readable, and well supported in Python, which is why it dominates modern APIs. XML is more verbose, but it supports features like attributes and namespaces, which some older systems rely on. CSV is simple and spreadsheet-friendly, but it handles nested data poorly, because it is flat by nature. Plain text is simplest of all, but it carries no structure, so the client must parse it manually, with care. Choosing a format, when an API offers a choice through an accept header, should be based on what your program needs, and on what it can parse reliably, rather than on habit alone.

Handling multiple formats in one client requires flexibility. Your client should check content type, then choose the appropriate parser, rather than hard-coding one parser for all responses. It should also handle the case where a format is unexpected, with a clear error, rather than a confusing parse failure. That flexibility, like other defensive patterns in this chapter, makes your client robust to API variations, and to server misconfigurations, where a response arrives in a different format than expected, due to an error page, or a wrong endpoint, both of which produce HTML, not JSON, as discussed earlier.

Finally, when you design your own API responses later, JSON is a sensible default, because it is widely understood, easy to parse, and expressive enough for most data. Offer other formats only if there is a clear need, such as CSV for spreadsheet users, or plain text for simple values. Each additional format adds maintenance cost, because it must be tested, documented, and kept consistent with the others. A single, well-documented format is usually better than several poorly maintained ones, a lesson that applies to API design as to program design generally, where simplicity and consistency beat variety for its own sake.

When an API offers a choice of format, test each format you intend to support, with real or fake responses, before relying on it. A format that looks simple in documentation may contain edge cases, such as special characters, nested structures, or empty values, that your parser must handle. Testing those edge cases early prevents failures later, when real data contains them, and your program, which was tested only with clean sample data, fails on messy real data. That testing habit applies to JSON as well, but it matters even more for less familiar formats, where your assumptions are more likely to be wrong.

Check: Why should an API client check content type instead of assuming every response is JSON?

Section 34: Handling Large Responses

Large API responses pose practical problems. A response containing thousands of items, or a large file, can consume significant memory if loaded entirely at once. It can also take a long time to download, increasing the chance of a timeout, or a network interruption, partway through. Handling large responses well therefore means thinking about size, time, and memory, before downloading, not after a failure. Pagination, discussed earlier, is one solution for large lists, because it splits them into manageable pages. Streaming is another, where data is processed as it arrives, in chunks, rather than waiting for the whole response, and then processing it at once.

Streaming suits data that can be processed incrementally, such as lines of a file, or records in a stream, where each piece can be handled independently, without needing the whole dataset at once. It reduces memory use, because only one chunk is held at a time, and it allows early progress, because processing starts before the download finishes, rather than waiting for completion. That early progress also helps users, who can see results appearing, rather than waiting in silence for a large download, a waiting that feels like a hang, even when the program is working correctly. Streaming, however, adds complexity, because errors may occur partway through, after some data has already been processed, and the program must handle partial results gracefully, rather than assuming success or failure is all-or-nothing.

A simpler approach, for moderately large responses, is to set appropriate timeouts, to use pagination, and to process data in batches, after downloading, rather than holding multiple copies in memory. Avoid keeping both the raw response and a transformed copy, and a displayed copy, all at once, if memory is limited, because that multiplies memory use unnecessarily. Process data once, transform it into the needed shape, and discard the raw form, unless it is needed for debugging, in which case log a summary, not the whole response, to avoid filling logs with huge entries, which, as discussed for logging, should be concise and safe.

Finally, consider whether a large response is necessary at all. Often, a large download results from requesting too much, such as all fields for all items, when only a few fields, or a filtered subset, are needed. Using query parameters to filter, to select fields, or to paginate, where the API supports them, reduces response size at the source, which is more efficient than downloading everything and discarding most of it locally. That efficiency, like other API habits in this chapter, benefits both client and server, by reducing bandwidth, time, and memory, for everyone, and it should be the first step, before streaming or batching, because the best way to handle a large response is often to avoid requesting it in the first place, by asking for exactly what is needed, no more.

Check: Why is requesting a smaller, filtered response often better than streaming a huge one?

Section 35: Tokens and Keys in Concept

API authentication often involves tokens or keys, which are secret strings that identify a client or a user. A key typically identifies an application, while a token often identifies a user, or a session, with specific permissions, and a limited lifetime, after which it expires and must be renewed. Those distinctions vary by API, and documentation should clarify them, but the general idea is the same: a secret string, presented with a request, proves that the requester is authorised, for the requested action. That proof must be protected, because anyone who obtains the secret can impersonate the authorised party, until the secret is revoked or expires, a risk that makes secret handling, as discussed earlier, a security priority, not a detail.

Tokens with limited lifetimes are safer than permanent keys, because exposure is time-limited: a leaked token stops working after expiry, while a leaked permanent key works until revoked, which may take time, if the leak is not noticed promptly. Short lifetimes, however, require renewal, which adds complexity, because the client must obtain a new token, before the old one expires, without interrupting the user, a renewal that, if handled poorly, causes failures, when a token expires mid-task, and the client does not renew it, or renews it incorrectly. Good clients handle renewal transparently, by checking expiry, and renewing proactively, before expiry, rather than waiting for a failure, and then renewing reactively, a proactive approach that, though it adds code, prevents interruptions, and improves user experience, by making authentication invisible, when it works correctly, rather than visible, through failures, when it does not.

Keys and tokens should never be shared between programs, or between users, unless the API's design explicitly intends sharing, because shared secrets make it impossible to tell who performed an action, and impossible to revoke one party's access, without revoking everyone's. Each program, and each user, should have its own credentials, with only the permissions it needs, a principle called least privilege, which limits harm, if a credential is leaked, because the leaked credential can only do what its owner could do, not everything, a limitation that, though it may seem restrictive, is actually protective, for users, whose data is exposed only to the extent of the leaked credential's permissions, rather than to full access, a protection that least privilege provides, by design, through, narrow permissions, per credential, a design that, providers, should support, by offering, scoped credentials, and that, clients, should use, by requesting, only needed permissions, rather than, full access, for convenience, a convenience that, though it simplifies, development, increases risk, unnecessarily, where narrow permissions, would suffice, and would, limit harm, if leaked, a harm limitation, that, professional API use, prioritises, over, convenience, for, credentials, generally, where, least privilege, is, a core security principle, for, all secrets, not just, API tokens, and that, this section, applies, specifically, to API authentication, where, scoped, short-lived, per-user credentials, are, safer, than, permanent, shared, full-access keys, a safety that, though it adds, complexity, in, renewal and scoping, pays off, in, reduced risk, and in, user trust, when, credentials, are handled, responsibly, from the start, for, every API, you use.

Check: Why are short-lived, narrowly scoped tokens safer than permanent, full-access keys?

Section 36: Configuration for API Settings

A program that talks to an API needs two kinds of settings, and keeping them apart is an important piece of design. Ordinary configuration includes values that are safe to share and to store in plain text: the base address of the service, a timeout value, the number of retries, or a default page size. These values describe how the program behaves, they are often the same for every user of the program, and they can reasonably be committed to version control, reviewed by colleagues, and discussed in documentation. They are part of the program's behaviour, and treating them as ordinary settings makes the program easier to understand and to adjust.

Secrets are different in kind, not merely in degree. A secret, such as an API key or a token, is a credential: whoever holds it can act as the program, or as the user the program serves, and its value must therefore be known to as few people and as few systems as possible. Storing a secret alongside ordinary configuration, in the same file and with the same handling, invites exactly the accidents that expose secrets: the file is committed to version control, copied to a colleague, included in a support bundle, or printed in a log, and the secret travels with it. Once a secret has entered version control, removing it from the current file does not remove it from history, and the only safe response is to revoke it and issue a new one, which is disruptive and embarrassing.

The practical separation is straightforward. Ordinary settings live in a configuration file, with clear names and sensible defaults, and they can be versioned and shared freely. Secrets live outside that file, supplied through the environment, or through a dedicated secrets mechanism, and referenced by name rather than by value. The program reads its settings from the file and its secrets from the environment, and the two sources are never mixed in a single committed document. A template configuration file can show the shape of the settings, with placeholder text where a secret would go, so that a new deployment can be set up without ever seeing a real credential in the repository.

import os
SETTINGS = {"timeout": 10, "retries": 3}
API_KEY = os.environ.get("API_KEY")
print(SETTINGS["timeout"], bool(API_KEY))

Keep API secrets separate from ordinary configuration settings, in a different source, with different handling, and never in a committed file. Ordinary settings can be shared, reviewed, and versioned; secrets must be supplied at run time, kept out of logs and repositories, and revoked promptly if they are ever exposed. That separation is a small structural decision, but it is the one that prevents the most common and most costly credential leak.

Check: Why should API secrets be kept separate from ordinary configuration settings?

Section 37: Mocking and Test Doubles Revisited

Mocks and test doubles let a program be tested without calling a real API, by standing in for the service with a fake that returns prepared responses. That is valuable: tests become fast, repeatable, and independent of network availability, and a test can exercise error cases, such as a failure response, that would be difficult to provoke from the real service. The danger of mocks is subtler than their benefit, and it grows over time. A mock describes the API as the programmer believed it to be when the mock was written, and the real API may since have changed: a field may have been renamed, a status code may have changed, or a response shape may have been extended. The mock, left unchanged, continues to return the old shape, the tests continue to pass, and the program continues to be wrong about the real service, with the test suite providing false confidence instead of protection.

Maintaining mocks therefore means treating them as a living description of the service, not as a fixture written once and forgotten. When the API's documentation changes, or when a new version is announced, the mocks should be reviewed and updated to match, and the program's handling of the new shape should be tested against the updated mock. Even better, the mocks should periodically be verified against the real API: a small set of checks that call the real service, in a safe environment, and compare its responses with what the mocks claim, can catch drift before it reaches users. That verification need not run on every test execution; it can run on a schedule, or before a release, as long as it runs regularly enough that drift is caught while it is still small.

The same discipline applies to the choice of double. A simple stub that returns a fixed response is appropriate when the program's logic does not depend on the service's behaviour beyond that response, while a more faithful fake, which simulates state or pagination, is appropriate when the program's logic interacts with the service over several calls. In either case, the double should be as simple as the test requires and no simpler: a double that is too simple may hide a real interaction, while a double that tries to reimplement the whole service becomes a second program to maintain, with its own bugs. Documenting what each double assumes about the service makes its maintenance easier, because the next reader can see which assumptions need checking when the service changes.

class FakeClient:
    def get_summary(self):
        return {"items": [1, 2], "total": 2}
print(FakeClient().get_summary()["total"])

Maintain your mocks as you would maintain any description of an external system: update them when the API changes, verify them against the real service on a regular basis, and record what they assume. A mock that is never checked is not a safety net; it is a photograph of a service that may no longer exist, and a test suite that passes against it proves only that the program agrees with its own photograph.

Check: Why must mocks be maintained and verified against real APIs, rather than left unchanged?

Section 38: Batching and Performance

Batching is the practice of sending several items in a single request, or of processing several items together, instead of making one request per item. It improves performance in an obvious way: fewer requests mean less overhead, fewer round trips, and less load on both the client and the service. A program that fetches a hundred records one at a time may spend most of its time waiting for responses, while the same program fetching them in batches of twenty spends a fraction of that time and leaves the service freer for other users. Batching also gives the service a chance to optimise its own work, since it can process a group of items together rather than handling each in isolation. For those reasons, batching is often the first performance improvement worth making when a program's API use grows.

Batching also changes the meaning of failure, and that change must be handled carefully. When a single request carries many items, some items may succeed while others fail: one record may be invalid, one operation may be rejected, or one item may exceed a limit that the others do not. A program that treats the batch as a single success or a single failure will get both cases wrong. Treating a partial failure as total success silently drops the failed items, leaving the program believing work is done that was never completed. Treating it as total failure, and retrying the whole batch, may duplicate the items that already succeeded, creating records twice or charging twice, depending on what the operation does. Neither outcome is acceptable, and both are avoidable.

The correct approach is to report and handle results per item. A batch response should be examined item by item, with each item's outcome recorded: succeeded, failed with a reason, or not attempted. Failed items can then be retried individually, reported to the user with their specific reasons, or set aside for later attention, without disturbing the items that succeeded. That per-item handling also makes the program's behaviour observable: a log that shows which items succeeded and which failed is far more useful than a log that shows only that a batch failed. The extra code required is modest, and it is the price of batching's performance benefit.

def handle_batch(results):
    done = [r for r in results if r["ok"]]
    failed = [r for r in results if not r["ok"]]
    return len(done), len(failed)
print(handle_batch([{"ok": True}, {"ok": False}]))

Batch when the work allows it, because fewer requests mean better performance for everyone, but never treat a batch as a single outcome. Report per-item results, retry only the items that failed, and make partial failure visible rather than hidden. A batching design that handles partial failure explicitly is one that can be trusted with real work, because its failures are precise, its retries are safe, and its users are told exactly what happened to each item they submitted.

Check: Why must batch API handling report per-item results instead of treating a batch as all success or all failure?

Section 39: Privacy and Data Minimisation

Privacy in API programs begins with a simple question: what data does this program actually need? Data minimisation is the practice of collecting only the data that is necessary for the program's purpose, and nothing more, and it protects users even when a program is never breached. A breach is not the only way data causes harm. Data that is collected can be misused by accident, disclosed through a bug, shared with a third party under terms the user never understood, or retained long after its purpose has ended and then used for a purpose nobody intended. Every field a program collects is a field it must protect, store, transmit, and eventually delete, and every unnecessary field is an unnecessary risk, carried for no benefit.

Minimisation also shapes how a program requests data from an API. An API may offer far more fields than a program needs, and requesting everything, because it is convenient, means the program holds sensitive details it never uses: full profiles when only a name was needed, precise locations when only a city mattered, or complete histories when only a recent summary was required. Requesting only the fields that serve the program's purpose, and discarding anything extra that arrives, reduces the program's exposure in the same way that collecting less reduces it. It also reduces the program's own storage and bandwidth costs, and it makes the program's behaviour easier to explain to a user who asks what data it holds and why.

The habit extends to retention and display. Data should be kept only as long as it is needed for its stated purpose, and then deleted, rather than accumulated indefinitely on the assumption that it might one day be useful. Data shown to a user, logged for debugging, or included in an error report should be the minimum needed for that purpose, with sensitive values omitted or masked. A log that contains a full user record is a privacy risk in itself, even if the log is never breached, because logs are copied, searched, and retained by people and systems that never needed the data. Designing with minimisation in mind means asking, at each point where data is collected, stored, shown, or logged, whether that data is truly needed there.

def minimise(record):
    return {"name": record.get("name"), "city": record.get("city")}
print(minimise({"name": "Ada", "city": "Lagos", "phone": "hidden"}))

Collect only the data your program needs, request only the fields it uses, keep data only as long as its purpose requires, and omit sensitive values from logs and displays. Those habits protect users not only against breaches, which may never happen, but against the everyday risks of misuse, accidental disclosure, and indefinite retention that accompany every unnecessary field. A program that holds little data has little to lose, and that is a stronger guarantee than any promise to protect a large collection well.

Check: Why does collecting only necessary data protect users, even if your program is never breached?

Section 40: Designing Your Own API Preview

Experience as an API consumer is the best preparation for designing an API for others, because it teaches, from the receiving side, what makes an interface pleasant or painful to use. A consumer who has struggled with vague error messages knows to write error responses that explain what went wrong and what the caller can do about it. A consumer who has been surprised by a field that disappeared without warning knows to version changes and to announce deprecations in advance. A consumer who has wasted time guessing at an undocumented behaviour knows to document not only the happy path, but the error cases, the limits, and the edge cases that real callers will meet. Designing an API preview, a small draft of an interface before it is built, is the place to apply those lessons deliberately.

A preview should start from the caller's point of view, not from the implementer's convenience. That means choosing clear, consistent names for resources and operations, so that a caller can guess the shape of an unfamiliar endpoint from the shape of a familiar one. It means deciding what a successful response contains, and what an error response contains, before any code is written, so that callers can write their handling once and rely on it. It means thinking about pagination, filtering, and ordering from the start, because adding them later, after callers have built around their absence, is disruptive. A preview that is reviewed by someone who will actually call it, even informally, will surface confusions that the designer, who already knows the intended meaning, cannot see.

The preview should also be honest about its limits. Stating the expected request rate, the maximum page size, and the fields that are guaranteed to be present prevents callers from depending on details that were never promised, and it gives the designer a contract to honour. Including example requests and responses, even for a small set of operations, is more useful than a long description, because examples show the shape of the data in a way that prose cannot. A preview is not a finished product, and it should be labelled as such, so that early feedback is welcomed rather than treated as a complaint about a finished design.

# A preview is a sketch, not an implementation:
# GET /items?page=2&limit=20
# -> {"items": [...], "total": 40, "page": 2}

Design your API preview from the caller's side, applying everything you learned as a consumer: clear names, consistent shapes, helpful errors, documented limits, and examples for every operation. Invite feedback on the preview before building, while changes are still cheap, and treat that feedback as the most valuable testing your interface will receive. An API designed that way is not merely functional; it is considerate, and considerate interfaces are the ones other programmers enjoy using and recommend to others.

Check: How does experience as an API consumer help you design a better API for others?

Section 41: Partial Failures in Batches

Partial failures happen when some items in a batch succeed, and others fail. That mixed result is harder to handle than a clean success or a clean failure, because the program must track which items need attention, and which are already done. A naive handler that treats any failure as a total failure may retry the whole batch, duplicating successful items. A naive handler that treats any success as a total success may silently drop failed items, losing data without warning. Both mistakes come from ignoring the per-item nature of batch results, and both are avoidable with explicit per-item handling, as discussed for batching generally.

Good batch handling starts with a clear result structure. For each item, record whether it succeeded, failed, or was not attempted, along with any error message for failures. That structure lets the program report accurately to the user, showing which items succeeded and which need fixing. It also lets retry logic target only failed items, rather than the whole batch, which is both efficient and safe, because successful items are not duplicated. That targeting requires keeping the original item list, and the result list, aligned, so that each result maps to the correct item, a mapping that should be tested, because misalignment, where a result is attributed to the wrong item, causes worse confusion than a simple failure, by marking wrong items as failed or successful.

Partial failures also raise a design question: should the program continue processing after a failure, or stop at the first failure? Continuing gives a complete picture, showing all failures at once, which helps users fix them in one pass. Stopping early saves time, when a failure indicates a systemic problem, such as authentication failure, that will affect all remaining items, making further attempts pointless. The right choice depends on the failure type: for independent item errors, such as invalid data for one item, continue, because other items may still succeed; for systemic errors, such as authentication or server failure, stop, because further attempts will fail for the same reason, wasting time and requests. Distinguishing those cases, in batch handling, mirrors the error-type distinction for single requests, and it makes batch programs both efficient and informative.

Finally, partial failures should be logged clearly, with per-item details, so that developers can diagnose patterns, such as a specific item type that always fails, or a specific error that affects many items. That logging, combined with user-facing summaries, ensures partial failures are neither hidden nor exaggerated: users see a fair summary, developers see full details, and retry logic sees exactly which items to retry. That three-way clarity, for users, developers, and retry logic, is the goal of partial failure handling, and it turns a complex mixed result into a manageable, actionable outcome, rather than a confusing mess, where no one knows what succeeded, what failed, or what to do next.

Check: Why should batch handling track results per item, and when should it stop at the first failure?

Section 42: Dates, Times and Timezones from APIs

APIs often return dates and times, and handling them correctly requires attention to format and timezone. A date string without timezone information is ambiguous: it could mean local time in the server's location, local time in the client's location, or universal time, and guessing wrongly shifts the time by hours, causing scheduling errors, or incorrect ordering of events. A well-designed API returns times in a standard format, with timezone information included, usually in universal time, which clients can then convert to local time for display, a conversion that, though it adds a step, removes ambiguity, because universal time is a single reference, from which all local times can be derived, correctly, if the conversion is done properly, with awareness of daylight saving, and of the user's actual timezone, not an assumed one. Parsing date strings should be done with care, not with manual string slicing, which is fragile, and which breaks when formats change slightly, such as a different separator, or an added timezone offset. If parsing fails, treat that as a validation failure, as discussed for API data generally, with a clear error, rather than proceeding with a wrong or default date, which could mislead users, who may act on an incorrect time, believing it is correct, a false confidence that validation prevents, by failing clearly, when a date cannot be parsed, instead of silently using a wrong date, which is worse, because it looks correct, while being wrong. Displaying dates also deserves thought. Users expect dates in their own timezone, and in a familiar format, not in universal time, or in the API's raw format, which may be precise but unreadable. Converting universal time to local time, for display, and formatting it clearly, with date, time, and, where relevant, timezone label, helps users understand when something happened, or will happen, without mental arithmetic, which is error-prone, especially across timezones, where mental conversion often goes wrong by an hour, due to daylight saving, or by a day, due to date line crossing, errors that proper conversion, with correct timezone data, avoids, by doing the arithmetic correctly, for the user's timezone, as configured, not as assumed, a configuration that should come from the user's settings, or from the system, rather than from a hard-coded timezone, which will be wrong for many users, a wrongness that, though it may seem minor, causes real confusion, for scheduling, deadlines, and event ordering, where an hour's error matters. Finally, storing dates from APIs should preserve timezone information, or store them in universal time, with a clear convention, documented in your code, so that later reads interpret them correctly, even if the program's timezone setting changes, or if the data is shared with another program, in a different timezone, both of which, without a clear storage convention, cause misinterpretation, where a stored local time is read as universal time, or vice versa, shifting it by hours, silently. A clear convention, such as always storing universal time, and converting only for display, prevents that, by making storage unambiguous, regardless of display timezone, a unambiguity that, like other conventions in this book, should be documented, and followed consistently, for all dates, in a program, so that dates remain trustworthy, across time, across timezones, and across programs, that share them.

Check: Why should API times be stored in universal time and converted only for display?

Section 43: Searching Through APIs

Searching through an API means asking the service to find matching items; you send a query; the service returns the items that match. This is different from listing all items and searching locally; listing everything works for a tiny dataset; it fails for a large one. Downloading a whole dataset wastes bandwidth; it also wastes time; it may hit a limit before the download finishes.

Server-side search is more effective as well as more efficient; the server can use indexes built for searching; it can rank results by relevance. It can match partial words and handle spelling variations, depending on the service; a local search over a partial download cannot match that. It can only search what you downloaded; if the full dataset is too large to download, local search will miss items; the server would have found those items in the full dataset.

A search request usually carries a query parameter; the parameter name depends on the service; common names include a query term or a search term. Read the documentation for the exact name; read it for the exact behaviour too; some services match whole words only; some match prefixes. Some rank by relevance and return the best matches first; some return results in a fixed order; knowing the behaviour changes how you present the results.

Search results are often paged, like list results; the first page holds the best or earliest matches; later pages hold the rest. Fetch only the pages you need; a user rarely needs hundreds of results at once; fetching them all wastes time and load. Show the first page clearly; offer a way to fetch more when the user asks; keep the query visible so the user knows what was searched.

Handle empty results gracefully; an empty result is a valid answer; it means nothing matched; do not treat it as an error. Show a clear message that explains the outcome; suggest a broader query or a different spelling when that helps; handle very broad queries with care too. A single common word may match a huge number of items; the service may limit such queries; your program should not request an enormous result set by default.

import json
import urllib.parse
import urllib.request

def search_items(base_url, query):
    params = urllib.parse.urlencode({"q": query})
    url = f"{base_url}/search?{params}"
    request = urllib.request.Request(url, headers={"Accept": "application/json"})
    with urllib.request.urlopen(request, timeout=10) as response:
        return json.loads(response.read().decode("utf-8"))

Test search with realistic queries; try a precise query that should match one item; try a broad query that matches many; try a query that matches nothing. Check that each case behaves sensibly; check that special characters in the query are encoded; an unencoded query can break the request or change its meaning. Server-side search is a service, not just a filter; use it for large datasets; send a clear query, page the results and present them honestly.

Check: Why is server-side search better than downloading data and searching locally?

Section 44: Authentication Flows in Concept

Many APIs protect access with tokens: short-lived credentials that a client presents with each request to prove it is authorised. A token has a lifetime, after which it expires, and a program that waits for expiry before acting will discover the problem at the worst possible moment: in the middle of a user's operation, when a request fails with an authentication error and the work that depended on it must be abandoned or repeated. Proactive renewal avoids that disruption by refreshing the token before it expires, so that the program always holds a valid credential when it needs one. The renewal is typically performed when the token is approaching its expiry, or when the program starts a new unit of work, rather than in response to a failure.

The benefit of proactive renewal is not only the avoidance of a failed request; it is the predictability it gives the program's behaviour. A program that renews early can handle renewal failures gracefully, for example by prompting the user to sign in again, or by pausing and retrying, before any user-visible operation has begun. A program that renews only after a failure must untangle the failed operation: decide whether it can be retried, whether any part of it already succeeded, and how to explain the interruption to the user. That untangling is error-prone, and it is work that proactive renewal makes unnecessary. Renewing early also spreads the cost of authentication over quiet moments, rather than concentrating it in the middle of busy ones.

Proactive renewal does not mean renewing constantly, which would waste requests and risk throttling. It means keeping track of when the current token expires, and refreshing it when the remaining lifetime falls below a sensible margin, with that margin chosen to cover the time a typical operation takes. The token's expiry should be treated as data the program manages, not as a surprise it reacts to, and the renewal logic should live in one place, in the client, so that every part of the program benefits from it. A failed renewal should still be handled, of course, but it should be handled as a planned event at the start of an operation, where the program can respond cleanly, rather than as an emergency in the middle of one.

class Token:
    def __init__(self, value, expires_in):
        self.value = value
        self.expires_in = expires_in
    def needs_renewal(self):
        return self.expires_in < 60
print(Token("abc", 30).needs_renewal())

Renew tokens proactively, before they expire, so that a program always holds a valid credential when it begins work. Track expiry as managed data, refresh with a margin that covers the operation ahead, and handle renewal failures at the start of an operation, where they can be dealt with cleanly. That habit turns authentication from a source of mid-operation failures into a quiet background task, and it makes the program's behaviour steadier for everyone who depends on it.

Check: Why is proactive token renewal better than waiting for an authentication failure and then renewing?

Section 45: Choosing an API

Choosing an API is a commitment: the program will depend on the service's availability, its data, its limits, and its future direction, and changing to a different service later is rarely cheap. That is why prototyping with real requests, before committing to an API for a significant program, is one of the most valuable steps in the selection process. Documentation describes an API as its authors intend it to be, but a prototype reveals it as it actually behaves: how quickly it responds, how its errors are phrased, whether its data matches the documented shape, and whether its limits allow the program's intended pattern of use. Those discoveries are inexpensive during a prototype, when nothing depends on the choice yet, and expensive afterwards, when the program has been built around assumptions that turn out to be wrong.

A useful prototype exercises the operations the program will actually need, not merely the simplest one. It should fetch the kind of data the program will display, perform the kind of search or filtering the program will rely on, and trigger at least one error, to see how failures are reported. It should also test the boundaries that matter for the program's scale: what happens when a result set is large, when a request is repeated, or when the program makes requests at the rate it will need in production. A prototype that only calls the happiest path confirms that the service exists; a prototype that exercises realistic use confirms that the service fits. Recording the results, including response times and unexpected behaviours, gives the selection decision an evidence base that a reading of the documentation alone cannot provide.

Prototyping also clarifies the non-technical fit. It shows whether the terms of use permit the program's intended purpose, whether the data's licence allows the intended redistribution, and whether the provider's support and status information give confidence in its reliability. It may reveal that a simpler, more limited API is a better fit than a powerful one whose complexity the program does not need, or that a local data source would serve the purpose without any external dependency at all. Those are decisions that are easy to make before building, and difficult to reverse afterwards, which is precisely why they belong in the prototyping stage.

# Prototype checklist, run against the real service:
# 1. Fetch realistic data. 2. Trigger an error.
# 3. Test a large result. 4. Check the rate limits.

Prototype with real requests before committing to an API, exercising the operations, errors, and scale the program will actually need. Record what you learn, compare it with the program's requirements, and let the evidence, not the documentation's promises, decide the choice. A few hours spent prototyping can save weeks spent rebuilding around an unsuitable service, and it gives the program a foundation that has already been tested against reality.

Check: Why should you prototype with real requests before committing to an API for a significant program?

Section 46: Documenting Your Own Client Use

API use involves decisions that are easy to forget and hard to reconstruct: why a particular endpoint was chosen, why a workaround was added, why a certain field is ignored, or why a request is retried in a particular way. Documenting those decisions when they are made, rather than later, preserves the reasoning while it is still fresh, and it saves the next person, who may be you in a few months, from having to rediscover it by experiment. A decision documented later is often documented inaccurately, because the original context has faded, the alternatives considered have been forgotten, and the documentation becomes a description of what the code does rather than an explanation of why it does it that way. Writing it down at the moment of decision captures the why, which is the part that cannot be recovered from the code alone.

The documentation need not be elaborate. A short note in the client module, or in the project's notes, stating the decision, the reason for it, and any relevant constraint, is usually enough. For example, a note might record that a particular workaround exists because the service returns an unexpected shape for one kind of record, with a link to the provider's documentation or issue, and a reminder to revisit the workaround when the service is updated. A note might record that a limit was chosen because the provider's terms allow a certain rate, or that a field is ignored because it is deprecated and will be removed. Each note is small, but together they form a record of the program's relationship with the service, which is exactly the knowledge that is lost when it is not written down.

Documenting at the time of decision also improves the decision itself. Writing down a reason forces a moment of reflection: is this workaround really necessary, is this limit really justified, is this choice really the best available? That reflection sometimes reveals a simpler alternative, or a question that should be asked of the provider before the workaround becomes permanent. It also makes review easier, because a reviewer can see not only what was changed, but why, and can challenge the reasoning rather than merely the code. Over time, the habit produces a codebase whose API use is explainable, which matters when the service changes, when a new team member joins, or when a failure must be diagnosed under pressure.

# NOTE: retry up to 3 times on a 503 response only.
# Reason: the provider documents 503 as temporary overload.
# Revisit if the provider changes its error guidance.
MAX_RETRIES = 3

Document API use decisions and workarounds when they are made, in a short note that records the decision, its reason, and any constraint or revisit condition. Keep those notes beside the code they explain, so they are found when they are needed. That habit preserves reasoning that code alone cannot preserve, and it turns a program's API use from a collection of unexplained quirks into a documented, reviewable, and maintainable part of the system.

Check: Why should API use decisions and workarounds be documented when they are made, not later?

Section 47: Monitoring API Usage

Individual request logs are useful for debugging a single failure: they show what was requested, what was returned, and how long it took. They are, however, a poor guide to the health of a program's API use as a whole, because they present one event at a time, without context. A program may log thousands of successful requests while slowly approaching a rate limit, or while its average response time quietly doubles, and no single log entry will reveal that trend. Monitoring usage trends, by aggregating requests over time, turns that stream of individual events into a picture that can be acted on: how many requests are made per hour, how the error rate changes, which operations are slowest, and how close the program comes to its limits.

Trend monitoring serves two purposes. The first is early warning. A rising error rate, a growing number of retries, or a response time that creeps upward over days are all signs of a problem that is developing, whether in the program, in the service, or in the network between them. Noticing such a trend early allows a fix to be made before users are affected, while noticing it only when requests start failing means the problem has already become an outage. The second purpose is planning. Knowing how usage grows, which operations dominate the load, and how close the program is to its quotas allows decisions about caching, batching, or renegotiating limits to be made on evidence, rather than on guesswork, and it allows capacity to be planned before it becomes urgent.

Effective monitoring does not require elaborate tooling. Counting requests by operation and by outcome, recording response times, and reviewing those counts regularly, even weekly, is enough to reveal the trends that matter. The review should ask simple questions: is usage growing, are errors increasing, is any operation approaching a limit, and is any operation unexpectedly slow? Those questions, asked regularly, catch most problems while they are still small. The results should also be kept in a form that can be compared over time, because a single week's numbers, without history, cannot show a trend, and it is the trend, not the snapshot, that carries the warning.

usage = {"requests": 120, "errors": 3, "slowest_ms": 850}
error_rate = usage["errors"] / usage["requests"]
print(f"error rate: {error_rate:.1%}")

Monitor API usage as trends over time, not only as individual log entries, by counting requests, errors, and response times by operation and reviewing them regularly. Trends give early warning of developing problems and evidence for planning decisions about caching, batching, and limits, while single logs, however detailed, can only explain a failure after it has happened. A program whose API use is monitored in this way is one whose problems are found early, understood in context, and fixed before they become outages.

Check: Why is monitoring API usage trends better than only reading individual request logs?

Section 48: When Not to Use an API

An API is a powerful tool, but it is not always the right tool, and recognising when a local solution is better is part of working well with APIs. The simplest case is data that already exists locally, or that changes rarely: if a program needs a list of countries, a set of conversion factors, or a reference table that is updated infrequently, storing that data locally, in a file or in the program itself, is simpler, faster, and more reliable than fetching it over the network on every run. A local copy works without a connection, responds instantly, and cannot be throttled, rate-limited, or changed without warning by a provider. The cost is that the local copy must be updated when the underlying data changes, but for stable data, that cost is small compared with the cost of a network dependency.

A second case is computation that does not need external data at all. If a program can perform a calculation, a transformation, or a lookup from data it already holds, doing so locally avoids the latency, the failure modes, and the privacy implications of sending data to a service. Sending user data to an external API, even for a simple operation, exposes that data to a third party, introduces a point of failure, and makes the program's behaviour depend on a service it does not control. When the local alternative is straightforward, the API adds risk without adding value, and the simpler program is the better program, even if the API version would also work.

A third case is scale and cost. A program that will make a very large number of requests, or that will be used by many people, may find that API limits, quotas, or charges make the external approach impractical, while a local solution scales without those constraints. Similarly, a program that must work offline, or in an environment where network access is unreliable, cannot depend on an API at all, and a local solution is not merely better but necessary. The decision should be made by comparing the whole cost of each approach, including reliability, privacy, latency, and ongoing maintenance, not merely by asking whether an API could technically do the job.

COUNTRIES = {"NG": "Nigeria", "GH": "Ghana", "KE": "Kenya"}
print(COUNTRIES.get("NG"))

Choose a local solution when the data is stable, when the work needs no external information, when privacy or offline operation matters, or when API limits and costs outweigh the benefit. An API is worth its dependency only when it provides something the program cannot reasonably provide itself: fresh data, a specialised service, or a capability that would be impractical to build locally. Asking that question honestly, before reaching for an API, keeps programs simpler, faster, and more reliable than they would be if every problem were solved by a network request.

Check: When is a local solution better than using an API, even if an API could technically do the job?

Section 49: Planning a Small API Project

A small API project begins, like any project, with a decision about what it is for, and stating that purpose in a single sentence is one of the most effective ways to keep the project on course. A clear purpose, such as fetching and displaying a daily summary from a public data service, gives every later decision a test: does this feature serve the purpose, or does it merely seem interesting? Features that pass the test are kept, and features that fail it are set aside, which prevents scope creep, the gradual addition of features that were not part of the original idea. Scope creep may feel like improvement, but in a small project it usually means delay, because each added feature brings its own design, testing, and debugging, and the project grows beyond what a learning exercise, or a small tool, needs.

A one-sentence purpose also guides the practical choices that follow. It determines which API to use, because the purpose names the data the program needs, and it determines which operations to implement, because the purpose names what the program must do with that data. It shapes the interface, because the purpose suggests what the user should see, and it shapes the testing, because the purpose defines what success looks like. Without that sentence, those choices are made in isolation, and the project can drift: the API is chosen for its novelty rather than its fit, the interface grows features that no user asked for, and the definition of done becomes unclear, so the project is never quite finished, even when it mostly works.

The sentence should be written before coding begins, and it should be specific enough to be useful. A vague purpose, such as building something with an API, provides no guidance, because almost any feature could be justified under it. A specific purpose, naming the data, the action, and the audience, provides a standard against which every addition can be judged. It should also be revisited when the project changes direction: if the purpose no longer describes what is being built, either the project should return to its purpose, or the purpose should be consciously revised, with the implications of the change understood. What should not happen is silent drift, where the project becomes something else without anyone deciding that it should.

# Purpose: fetch and display a daily summary from a public data service.
# Features that serve this purpose: fetch, summarise, display.
# Features that do not: user accounts, charts, alerts.

Begin a small API project by writing its purpose in one sentence, naming the data, the action, and the audience, and use that sentence as the test for every feature that is proposed afterwards. A project guided by a clear purpose finishes, because its scope is bounded, its choices are coherent, and its definition of done is known from the start. A project without one may grow impressively, but it is far less likely to finish at all, and finishing is the point of a small project.

Check: Why does a clear one-sentence purpose help a small API project finish successfully?

Section 50: Habits for Working with APIs

The habits in this chapter form a workflow that is worth carrying into every API project, because together they cover the whole life of an API interaction, from first reading to long-term maintenance. The workflow begins before any code is written, with reading the documentation and the terms of use, and with prototyping real requests to confirm that the service fits the program's needs. It continues with design: a centralised client that hides the details of requests and responses, internal data models that insulate the rest of the program from the service's shapes, and validation of everything that arrives, on the assumption that external data cannot be trusted merely because it came from a documented source. Those early habits prevent the most common failures, because they are applied when change is still cheap.

The workflow continues during operation, with habits that keep the program reliable and considerate. Check the status of a response before parsing it, set timeouts on every request, retry failures with a pause rather than immediately, and respect rate limits and terms, caching where caching is allowed. Report results per item when working in batches, so partial failure is visible and manageable, and renew tokens before they expire, so authentication never interrupts a user's work. Handle errors gracefully, with messages that explain what happened and what the user can do, rather than with crashes or silent failures. Each of these habits is small on its own, but together they make a program that degrades gracefully when conditions worsen, instead of failing suddenly when they change.

The workflow ends with maintenance, which is where many API programs are weakest. Keep mocks updated and verify them against the real service, document decisions and workarounds when they are made, monitor usage trends rather than only individual logs, and minimise the data collected and retained, so that privacy is protected by design rather than by promise. Revisit the program's API choices as the service evolves, and be willing to replace a service, or to move to a local solution, when the fit no longer holds. A program maintained in this way stays correct as its dependencies change, which is the real test of API work, since services change far more often than the programs that use them are rewritten.

# Workflow: read docs, prototype, centralise, validate,
# handle status first, respect limits, monitor trends.

Carry this workflow into every API project: read and prototype first, centralise and validate by design, handle failures gracefully during operation, and maintain, document, and monitor afterwards. No single habit in the workflow is sufficient on its own, but together they form a practice that is reliable, considerate, and maintainable, and that is the standard this chapter has aimed to teach. A program built on those habits will not be free of problems, but its problems will be understandable, its failures will be graceful, and its users will be able to trust it.

Check: What workflow habits from this chapter should you carry into every API project?

Practical Task

Build a small API client in Python, using only the standard library or a common HTTP client you already use, for a public, documented API of your choice that does not require payment. Read its documentation first, then design a centralised client with methods for at least two operations, such as listing items and fetching one item. Implement status-first response handling, validation of response data, timeouts, and graceful error messages for at least three error types, including a not-found error and a server error. Test the client with fake responses for success, error, pagination, and invalid data cases, and run at least one real request at low volume, respecting rate limits. Write a short note documenting which endpoints you used, what rate limits apply, and how your client respects them, and keep any credentials, if the API requires them, out of your code and logs, using a secure, approved method for test credentials only.

Chapter Questions

1. Why should you check a response's status code before parsing its body, and what can go wrong if you do not?
2. How do pagination, rate limits, and caching interact in a client that retrieves large lists regularly?
3. When should you retry a failed API request, and when should you correct the request or ask the user instead?
4. Why is a centralised API client with internal data models better than scattering raw API data use throughout a program?
5. What makes an API client responsible in its use of a shared service, beyond merely working correctly when the service succeeds?