Chapter 31: Automation and Scripting

Section 1

Automation means letting a program carry out a task that a person would otherwise repeat by hand, step by step, with the same care every time. When you automate, you describe the task once, in code, and then the computer can run that description whenever it is needed. The description might be short, such as copying files from one folder to another, or longer, such as gathering data from several files, tidying it, and producing a summary. The important shift is not speed alone, although speed often follows. It is reliability. A careful script does not get tired, distracted, or forget the third step on a busy afternoon, and that consistency is often worth more than raw speed.

A good way to think about automation is to imagine a checklist that can run itself. If you can write down the steps clearly enough for someone else to follow them without asking questions, you are close to being able to write them as a script. If the steps contain vague judgements, such as deciding whether a photograph looks pleasant, automation becomes harder, because the program needs a rule it can apply. That does not mean such tasks can never be automated, only that they need clearer rules, or a human step left in the middle. Many real workflows are partly automated: the machine does the repetitive gathering and sorting, and a person makes the final judgement.

Automation appears in almost every area where computers are used. A teacher might generate a set of worksheets from a list of questions. A shopkeeper might turn a day of sales records into a weekly total. A researcher might rename hundreds of data files so that they sort in date order. None of these tasks requires a large application with buttons and menus. They require a small, trustworthy program that does one job well. Learning to build such programs is therefore a practical skill, even for people who never intend to become full-time software developers.

It helps to start small. Choose a task you already understand, that you do regularly, and that has a clear result you can check. Automating a task you do not understand is risky, because you may faithfully reproduce a mistake at a much larger scale. Automating a task you do once a year may not repay the effort of writing and testing the script. The sweet spot is a task that is boring, repeatable, and easy to verify. When the script finishes, you should be able to look at its output and say, with confidence, whether it did the right thing.

Finally, automation is not about removing people from work. It is about moving human attention to the parts of work that need judgement, creativity, and care, while the machine handles the parts that are merely repetitive. A well-built script is like a reliable colleague: it does what it was asked to do, reports what it did, and stops politely when something unexpected happens instead of guessing. Building that kind of reliability is the theme of this chapter.

Check: What makes a task a good candidate for automation?

Section 2

Not every repeated task should be automated, and learning to choose well is part of the skill. A useful test is to ask three questions. First, how often does the task happen? Second, how long does it take by hand, and how error-prone is it? Third, how stable are the steps? A task that happens every day, takes an hour, and follows the same steps each time is an excellent candidate. A task that happens once, takes five minutes, and changes every time is usually better done by hand. Between those extremes lies a judgement call, and experience helps you make it more quickly.

Frequency matters because writing a script costs time too. If a manual task takes ten minutes and you do it twice a year, spending three hours automating it does not save time in any practical sense. It might still be worth doing if the task is stressful, if mistakes are costly, or if you want to learn from building the script. Those are legitimate reasons, but they should be named honestly. A script built for learning does not need the same polish as a script that will run every night without supervision. Being clear about the purpose keeps the work proportionate.

Error-proneness is often a stronger reason than speed. Copying numbers from one file to another, renaming files by hand, or reformatting dates are all tasks where a small slip can cause confusion later. A script that applies the same rule to every item removes that class of error, provided the rule itself is correct. That condition matters. Automation does not remove mistakes; it moves them. Instead of making a mistake on one file, you might make the same mistake on a thousand files if the rule is wrong. Testing on a small sample before running a script on everything is therefore essential.

Stability is the third question. If the steps change every week because the people involved keep changing their minds, a script will need constant updating, and the maintenance may cost more than the manual work saved. In such cases, it can help to automate only the stable core and leave the changing parts to a person. For example, a script might gather and sort data, while a person decides how to present the final summary. Partial automation is often the wisest choice, and it should not be seen as a failure.

There is also a human factor. Some tasks are worth doing by hand because doing them teaches you something, or because the slow pace gives you time to notice problems. Reading through a set of records by hand might reveal an odd pattern that a script would pass over. Automation should support attention, not replace it entirely where attention is the point. Asking whether the task teaches you something, or whether its slowness protects quality, can save you from automating away a valuable check.

Check: Why might a frequently repeated task still not be worth automating?

Section 3

People often use the words script and program as if they mean the same thing, and in everyday speech they largely do. Both are code that a computer runs. The difference is mostly about size, purpose, and how the code is used. A script is usually a short, focused piece of code written to do one job, often run from a command line, and often written quickly to solve a problem at hand. A program, in the larger sense, might have a graphical interface, many features, settings, and users who never see its code. The boundary is soft, and a script can grow into a program over time.

Scripts tend to be linear. They start at the top, do a series of steps, and finish. They may read files, transform data, and write results. They rarely need to stay running for hours, respond to mouse clicks, or manage many users at once. That simplicity is a strength. A short script is easier to read, test, and fix than a large application, and it can be written and changed in an afternoon. Many professional developers keep a collection of small scripts for everyday jobs, even when their main work involves much larger systems.

Another difference lies in who runs the code and how. A script is often run by the person who wrote it, or by a scheduler on a computer they control. It may assume that certain folders exist, or that input files follow a particular pattern. A widely distributed program, by contrast, must cope with many different computers, users, and unexpected inputs. That is why scripts can be informal in ways that published software cannot. Informal does not mean careless, however. A script that deletes or overwrites files still needs care, even if only one person runs it.

It is useful to think of scripts as tools rather than products. A hammer is not judged by whether it has a beautiful interface; it is judged by whether it does its job safely and predictably. The same applies to a script. Clear output, a sensible response to missing files, and a way to stop safely are more valuable than extra features. If a script grows so large that it needs its own manual, it may be time to treat it as a program, with proper structure, tests, and documentation. That transition is natural, and recognising it early prevents a tangle.

Finally, the skills transfer in both directions. Writing good scripts teaches you to break problems into steps, handle errors, and organise code into functions. Those skills are exactly what larger programs need. Conversely, habits learned from larger programs, such as naming things clearly and keeping functions short, make scripts easier to maintain. Do not think of scripting as a lesser activity. It is a core part of practical programming.

Check: How does a script typically differ from a large application?

Section 4

Running a Python script is straightforward once you know the basic pattern, but a few details are worth getting right from the start. You write your code in a text file ending in .py, and you run it with the Python interpreter from a terminal or command prompt. The interpreter reads the file from top to bottom and carries out each instruction. If the file contains an error, the interpreter stops and prints a message describing what went wrong and where. Reading that message carefully is one of the most valuable habits a beginner can build, because it usually points directly at the problem.

Before running a script, it helps to know which folder you are in. The terminal has a current working directory, and relative file paths in your script are understood relative to that directory, not necessarily relative to where the script file lives. This is a common source of confusion. A script that works when you run it from one folder may fail when you run it from another, simply because it cannot find its input files. Using clear paths, or printing the working directory while you are learning, can remove much of this mystery. Over time, you will learn to write scripts that are explicit about where they look for files.

It is also wise to run a new script in a safe setting first. If the script creates, renames, or deletes files, test it on copies in a practice folder, not on your real documents. That habit costs a few minutes and can save hours of recovery. Even experienced developers test file-changing scripts on sample data first. The goal is not to be fearful, but to make mistakes cheap. A mistake in a practice folder is a lesson; the same mistake in a folder of irreplaceable work is a crisis.

When a script runs, it produces output. Some output goes to the screen, and some goes into files. Screen output is useful while you are developing, because you can watch what happens. For scripts that run unattended, writing a short log to a file is more useful, because nobody is watching the screen. A simple message such as how many files were processed, and whether anything was skipped, turns a silent script into a trustworthy one. You will return to logging later in this chapter, but the principle starts here: a script should tell you what it did.

Finally, remember that running a script twice should ideally be safe. If the first run creates output files, the second run should either overwrite them cleanly, skip work that is already done, or ask what to do. Scripts that pile up duplicate files, or fail because an output file already exists, are frustrating to use. Thinking about the second run, not just the first, is a mark of a mature script writer.

Check: Why does the terminal's current working directory matter when running a script?

Section 5

Most automation works with files and folders, so becoming comfortable with the filesystem is essential. A filesystem organises data into folders, which can contain files and other folders, forming a tree. Every file has a path, which is the route from the root of the tree to that file. Python provides tools in its standard library for working with paths, listing folders, checking whether something exists, and creating new folders. Using those tools, rather than typing commands by hand, is what makes file automation possible.

One of the first lessons is to distinguish between a path and a file. A path is a description of a location; it does not guarantee that anything is actually there. A script should check before it acts, especially before it overwrites or deletes. Checking that an input folder exists, and that it contains the files you expect, prevents a script from failing halfway through, or worse, from creating empty output that looks like success. A short check at the start, with a clear message if something is missing, saves a great deal of debugging later.

Listing the contents of a folder is a common starting point. A script might list every file in a folder, filter the list to only those with a certain ending, and then process each one in turn. This pattern, list, filter, process, appears in countless automation tasks. It is worth learning well, because once you can do it reliably, you can adapt it to renaming, converting, summarising, and archiving. The filtering step is important: processing every file in a folder, including ones you did not intend, is a common source of accidents.

Creating folders is another basic operation. A script that organises files by year, or by type, will need to create destination folders if they do not already exist. Python can do this safely, creating parent folders as needed, and doing nothing harmful if the folder already exists. The key is to build destination paths carefully, joining folder names with the proper tools rather than gluing text together with slashes. That care keeps the script working on different operating systems, where path separators differ.

Finally, treat file operations as actions with consequences. Reading a file is safe. Writing, moving, renaming, and deleting are not, in the sense that they change something that may be hard to undo. Good scripts therefore separate the deciding step from the acting step. First, decide what would change. Then, if possible, show that plan or write it to a log. Only then make the changes. That separation is the foundation of the dry-run idea you will meet later in this chapter.

A script that only prints filenames while you watch it is a good first step, but a script that records its decisions in a file is far easier to review later, especially when hundreds of files are involved.

Check: What is the list, filter, process pattern used for in file automation?

Section 6

Batch renaming is a classic automation task, and a good illustration of both the power and the risk of scripts. Suppose you have a folder of photographs with names assigned by a camera, and you want names that include the date and a short description so that they sort sensibly. Renaming them one by one would take a long time and invite typos. A script can apply the same naming rule to every file in seconds. The rule might add a prefix, replace spaces with underscores, number files in order, or insert a date taken from the file itself.

The first step in a safe renaming script is to decide the rule precisely, and to test it on paper or on screen before renaming anything. A useful technique is to print pairs of old and new names, without actually renaming, and to read through them. Does the numbering start where you expect? Do any two files end up with the same new name? Are special characters handled sensibly? Reading the planned changes catches most problems while they are still cheap to fix. Only when the plan looks right should the script perform the renames.

Collisions deserve special attention. If two different files would receive the same new name, renaming the first would overwrite or block the second, depending on how the script is written. A careful script checks for collisions before it starts, and stops with a clear message if it finds any. It might also refuse to rename a file onto an existing file. These guards feel like extra work when everything is going well, but they are exactly what protects you on the day when something is unusual.

Ordering matters too. If you number files, the order in which the computer lists them may not be the order you expect. Sorting by name, by date, or by another clear rule gives a predictable result. Without an explicit sort, the order may depend on the operating system, and running the same script on another computer could produce different numbers. Stating the sort rule in the code, and perhaps in a comment, makes the behaviour repeatable and understandable to someone reading the script later.

Finally, keep a record. A renaming script that writes a small log, mapping old names to new names, gives you a way back if you change your mind. Even better, test the script on copies first. Renaming is a small change to each file, but across hundreds of files it is a large change to your collection. The habits in this section, plan first, check collisions, sort explicitly, and log the result, apply to almost every file-changing script you will write.

If you are unsure about a naming rule, rename a small test set first and live with the result for a day before applying the rule to the whole collection.

Check: Why should a renaming script print its planned changes before renaming anything?

Section 7

Text processing is at the heart of many automation tasks, because so much information is stored as text. Logs, reports, lists, configuration files, and exported data are all text, and Python is well suited to reading, searching, and reshaping it. A typical text automation reads a file line by line, looks for lines that match a pattern, extracts the useful parts, and writes a new file or a summary. Working line by line keeps memory use low, even for large files, and it matches how people naturally think about text: one line, one record, one idea.

Before processing text, it is worth understanding its encoding. Text files store characters as numbers, and the most common modern encoding is UTF-8. Most of the time, Python handles this sensibly by default, but files created by older programs or on other systems may use a different encoding, and opening them with the wrong one can produce errors or strange characters. If you meet such a file, the fix is usually to specify the correct encoding when opening it, or to convert the file once, carefully, rather than guessing repeatedly. Keeping your own output in UTF-8 is a safe default for new files.

Searching text is often done with simple tools first. Checking whether a line contains a word, starts with a prefix, or matches a pattern may be enough. More complex patterns can be handled with regular expressions, which are powerful but easy to overuse. A good rule is to start with the simplest tool that works, and to reach for regular expressions only when the pattern genuinely needs them, such as matching dates in several formats. Simple code is easier to read, test, and fix, and for automation that matters more than cleverness.

When you transform text, keep the original unchanged until the new version is complete and checked. Writing output to a new file, rather than overwriting the input in place, gives you a fallback if something goes wrong. If the output looks correct, you can then replace the original, or keep both. This caution is especially important for scripts that clean data, because cleaning can remove information that later turns out to have been valuable. A script that reports how many lines it changed, and why, helps you judge whether the cleaning behaved as intended.

Finally, think about line endings and whitespace. Different systems mark the end of a line differently, and stray spaces or blank lines can confuse later processing. Python's text handling usually smooths over these differences, but when you compare or count lines, it helps to strip trailing whitespace deliberately rather than assuming it is absent. Small details like these are what separate a script that works on your test file from one that works on real data.

Check: Why is it safer to write transformed text to a new file instead of overwriting the input?

Section 8

Comma-separated values, usually called CSV, are one of the most common formats for exchanging tabular data. A spreadsheet can export a table as CSV, and many programs can read it back. Each line in a CSV file represents a row, and commas separate the columns, although fields that contain commas are usually wrapped in quotes. Python's standard library includes a dedicated CSV module, and using it is far safer than splitting each line on commas by hand, because the module understands quoting, embedded commas, and line breaks inside fields.

A typical CSV automation reads a file, processes each row as a list or dictionary, and writes a new file or a summary. Using dictionaries, where each column is named by its header, makes code easier to read than using numbered positions, because row["price"] says more than row[3]. It also makes the code less fragile if columns are reordered, although it still depends on the header names being spelled as expected. Checking the header row at the start, and reporting a clear error if an expected column is missing, prevents confusing failures later.

Data in CSV files is almost always text, even when it represents numbers or dates. A script that adds up a column must therefore convert text to numbers, and it must decide what to do with empty cells, stray spaces, or values that do not convert. Skipping bad rows silently is dangerous, because the resulting totals will be wrong without any warning. Better approaches include counting skipped rows, writing them to a separate file for review, or stopping with a clear message. The right choice depends on the task, but silence is rarely the right choice.

Writing CSV files needs similar care. The CSV module handles quoting for you, so fields containing commas, quotes, or line breaks are written correctly. Opening the output file with the correct newline handling, as recommended in Python's documentation, prevents extra blank lines on some systems. It is a small detail, but it is the kind of detail that makes a script's output usable in a spreadsheet without manual repair. Testing the output by opening it in a spreadsheet, not just by looking at the raw text, is a worthwhile final check.

CSV automation is a good place to practise the habits of this chapter: read a sample, validate headers, handle bad data visibly, and keep the original file untouched. Once you can summarise a CSV file reliably, you can adapt the same pattern to reports, data cleaning, and combining several files into one. The format is simple, but treating it with respect avoids a whole class of quiet errors.

When a CSV file comes from another program, open a small sample and inspect its headers and quoting before writing code that assumes a particular layout.

Check: Why should you use Python's CSV module instead of splitting lines on commas by hand?

Section 9

Scheduling is the idea of running a script automatically at a chosen time, without a person starting it by hand. A script that produces a morning report, backs up a folder each night, or checks a folder for new files every hour becomes far more useful when it runs on a schedule. Different operating systems provide different schedulers, but the concept is the same: you tell the system which script to run, when to run it, and what should happen to its output. The script itself does not need to know about the schedule; it simply does its job when it is started.

A scheduled script has stricter requirements than a script you run yourself. When you run a script by hand, you are there to notice an error, answer a question, or type a missing filename. A scheduled script has no one watching. It must therefore work without interaction, use absolute paths or paths it can determine reliably, and handle missing input gracefully. It should not wait for keyboard input, open windows, or assume that a particular person is logged in. Designing for unattended running is a mindset shift, and it is worth adopting even for scripts you currently run by hand.

Logging becomes essential for scheduled scripts. Since nobody watches the run, the script must leave a record of what it did, including the time, how many items it processed, and any problems it met. That log is how you discover that a script has been failing quietly for a week, or that its input changed. A log that only records successes is less useful than one that also records warnings and skips. Checking the log occasionally is part of maintaining an automated workflow, not an optional extra.

Another consideration is what happens if a scheduled run takes longer than expected and overlaps with the next run. Two copies of the same script writing to the same output file can corrupt the result. Simple safeguards include writing to a temporary file and then moving it into place, or using a lock file to show that a run is already in progress. For many small tasks, these safeguards are unnecessary, but for scripts that run frequently or take a long time, they are worth considering before a problem occurs.

Finally, keep scheduled scripts simple and testable. You should be able to run the same script by hand and get the same result, so that you can debug it without waiting for the next scheduled time. If a script only works when the scheduler runs it, it will be hard to fix. A script that is easy to run by hand, writes a clear log, and needs no interaction is ready to be scheduled with confidence.

Check: Why must a scheduled script avoid waiting for keyboard input?

Section 10

A script that runs without telling you what it did is hard to trust. Logging is the practice of recording, in an organised way, what a script did, when it did it, and what went wrong. Good logs are not just for debugging. They are how you prove, days later, that a task ran, how long it took, and which items were skipped. For automation, where the whole point is that nobody watches the run, a log is the script's way of reporting back.

Python's standard logging module is a good tool for this, but even simple print statements written to a file can serve as a basic log. The important thing is consistency. Each run should record its start time, its main steps, and its finish, in a format that is easy to scan. A log that mixes chatty detail with important warnings is hard to use, so it helps to distinguish levels: information about normal progress, warnings about skipped items, and errors about failures. Even if you use simple prints, marking warnings and errors clearly helps a reader find problems quickly.

What should a log contain? At minimum, the time of the run, the input it used, the number of items processed, and the number skipped or failed. If items were skipped, the log should say which ones and why, or point to a separate file listing them. A log that says only that something failed, without saying what, forces you to rerun the script to find out. A log that says that a particular file was skipped because it was empty, by contrast, lets you decide at a glance whether that matters.

Log files themselves need managing. A script that runs every day and appends to the same log will eventually create a very large file. Rotating logs, which means starting a new file periodically and keeping only a limited number of old ones, solves this. Some schedulers and logging tools handle rotation for you. Even a simple approach, such as including the date in the log filename, keeps files manageable and makes it easy to find the log for a particular day.

Finally, remember that logs can contain sensitive information. If a script processes personal data, its log should not record that data in full. Recording counts, filenames, or error types is usually enough, and it avoids creating a second copy of sensitive information in a file that might be less protected. Thoughtful logging balances usefulness with restraint, and that balance is part of writing scripts that others can trust.

A short summary at the end of a log, stating how many items succeeded and how many were skipped, makes a quick daily check far easier than reading every line.

Check: What minimum information should an automation log record for each run?

Section 11

Error handling is what separates a script that is merely useful from one that is trustworthy. When a script runs by hand, an error stops it and the person running it can investigate. When a script processes hundreds of files, a single bad file should not necessarily stop the whole job, nor should it be silently ignored. The right response depends on the task. Sometimes stopping is safest, especially when later steps depend on earlier ones. Sometimes skipping the bad item, recording it, and continuing is better, because the rest of the work is still valuable.

Python's exception mechanism gives you control over this choice. You can wrap a risky step, such as opening a file or converting a value, in a try block, and decide what to do if it fails. The key is to catch only the errors you expect and understand. Catching every possible error and continuing regardless can hide serious problems, such as a full disk or a permissions fault, and produce output that looks complete when it is not. A narrow, thoughtful catch, paired with a clear log message, is far better than a broad catch that swallows everything.

It also helps to distinguish between errors you can recover from and errors you cannot. A missing optional file might be recoverable: the script can use a default or skip that step. A missing required input file is not recoverable: the script should stop promptly with a message naming the file it expected. Continuing after a required input is missing usually produces misleading output, such as an empty report that looks like a day with no activity. Failing early, with a clear message, is a kindness to the person who will read the log.

Another useful habit is to validate inputs before processing begins. Check that required folders exist, that files are readable, and that configuration values are sensible. This early validation catches the most common problems before any work is done, which means the script either runs cleanly or fails before it has changed anything. That is much easier to recover from than a failure halfway through, when some files have been changed and others have not.

Finally, test your error handling deliberately. Create a test folder with a missing file, an empty file, and a file with unexpected content, and run the script against it. Watching how the script responds teaches you more than reading about exceptions ever will. A script whose error paths have been tested is a script you can leave running with confidence.

When a script processes a list, it can also collect failures into a separate report instead of stopping at the first problem, which lets you fix several bad items in one pass and rerun only the failed ones, provided the successful work is clearly recorded and not duplicated.

Check: Why is catching every possible error and continuing usually a bad idea?

Section 12

Idempotence is a long word for a simple and valuable idea: running a script twice should give the same result as running it once. A script that creates a report file is idempotent if running it again simply replaces the report with an identical copy. A script that appends to a report each time it runs is not idempotent, because running it twice doubles the content. For automation, especially scheduled automation, idempotence is a safety net. If a run is repeated by accident, or if you rerun a script after fixing a problem, the result should still be correct.

Achieving idempotence often means thinking about how output is written. Overwriting an output file, rather than appending to it, is the simplest approach for reports and summaries. For organising files, it means that a file already in its correct place is left alone, rather than moved again or duplicated. For creating folders, it means using an operation that succeeds whether or not the folder already exists. These small choices add up to a script that can be rerun without fear.

Idempotence also helps with recovery. Suppose a script fails halfway through processing a large set of files. If the script is idempotent, you can simply run it again, and it will redo the unfinished work without duplicating the finished work. If it is not idempotent, recovery becomes a careful manual job: you must work out which files were already processed and which were not, and you must clean up any duplicates. Designing for reruns therefore saves effort not only on the happy path, but also when things go wrong.

There are limits. Some actions cannot truly be undone or repeated safely. Sending a message, charging a payment, or deleting a file are not idempotent in the same way, because repeating them has real effects. For such actions, a script needs stronger safeguards, such as recording what it has already done and checking that record before acting again. In this chapter, the focus is on file and data automation, where idempotence is usually achievable with simple techniques, and those techniques are worth learning well.

A practical test is to run your script twice on the same test data and compare the results. The output files should be identical, the folder structure should be unchanged by the second run, and the log should show that the second run found work already done or safely replaced it. If the second run creates duplicates or errors, the script is not yet idempotent. Fixing that, before the script runs on real data, is time well spent.

A script that downloads a file, for example, can check whether the file already exists and is complete before downloading it again, which saves time and bandwidth and avoids replacing a good file with a partial download if the network fails midway. This kind of guard makes repeated runs cheap and predictable.

Check: What does it mean for a script to be idempotent?

Section 13

A dry run is a rehearsal: the script goes through all its steps, decides what it would do, and reports its plan, without actually changing anything. This is one of the most valuable safety features a file-changing script can have. It lets you review a plan that affects hundreds of files before committing to it, and it turns a risky operation into a calm review. Many experienced developers will not run a renaming, moving, or deleting script without a dry run first, even when they wrote the script themselves.

Implementing a dry run can be simple. A script might accept a flag, such as a command-line option, that switches it into dry-run mode. In that mode, wherever the script would normally rename, move, or delete, it instead prints a message describing the action. The decision logic, the filtering, and the sorting all run exactly as they would in a real run, so the dry-run output is a faithful preview. The only difference is that the final action is replaced by a message. Keeping the two modes close together in code ensures that the preview really matches what would happen.

Dry runs are also useful for testing. When you change a script, running it in dry-run mode on sample data shows you the effect of the change without risk. When you run it on real data for the first time, a dry run lets you spot surprises, such as files you did not expect to match your filter, or a destination folder you mistyped. Reading a dry-run report carefully is a skill in itself: look for counts that seem too high or too low, for files that should not be there, and for actions that seem out of place.

It is important that dry-run mode is obvious. The script should print a clear banner, such as a message that it is running in dry-run mode and making no changes, at the start and the end of its output. A dry run that looks identical to a real run can mislead a reader into thinking changes were made when they were not, or worse, into thinking no changes were made when they were. Making the mode unmistakable in both the output and the log prevents that confusion.

Finally, do not treat a dry run as a substitute for backups. A dry run reduces the chance of a mistake, but it cannot remove it entirely, because a real run may still meet conditions that the dry run did not. The two practices work together: dry run to review the plan, backup to recover if the plan still proves wrong. Together, they make file automation far safer than doing the same work by hand.

Check: How does a dry run differ from a real run of the same script?

Section 14

Before a script changes files, making a backup is a simple precaution that can save a great deal of trouble. A backup is a copy of the original data, kept somewhere safe, so that if the script's changes prove wrong, the original can be restored. This sounds obvious, yet it is often skipped because the script seems simple, or because the data seems unimportant at the time. The value of a backup only becomes clear when something goes wrong, and by then it is too late to create one.

A backup does not need to be elaborate. For a folder of files, copying the folder to a new location, with the date in its name, is often enough. For a single file, copying it to a nearby file with a suffix such as .bak, or into a backup folder, may suffice. The important qualities are that the backup is complete, that it is made before the script runs, and that you know where it is. A backup that is incomplete, made after the changes, or hidden in a forgotten location is little better than no backup at all.

Automation scripts can create their own backups, and for scripts that run regularly, that is often the best approach. A script might copy its input folder to a dated backup folder before it begins processing, and then log the backup location. That way, every run leaves a trail back to the original data. Over time, however, backups accumulate, and old ones should be removed on a sensible schedule, or they will fill the disk. Keeping, for example, the last few backups, and deleting older ones, balances safety against storage.

There is a subtle point about testing backups. A backup that has never been restored is only a hope, not a guarantee. Occasionally, restoring a backup into a test folder and checking that the files are intact and complete turns that hope into confidence. This is especially important for scripts that run unattended: if the backup step itself is failing quietly, you will only discover it when you need the backup most. A log message confirming that the backup was created, and how many files it contains, is a small but useful check.

Finally, remember that backups protect against mistakes, not against every disaster. A backup on the same disk does not help if the disk itself fails. For truly important data, a copy on another device or in another location is needed. For everyday scripting, however, a local, dated copy before a risky run is a proportionate and highly effective habit.

If a script will change many files, naming the backup folder with the current date and time makes it easy to tell which backup belongs to which run, and to restore the right one without guessing among folders with similar names.

Check: Why should a backup be made before a script runs, not after?

Section 15

Working with paths is easier and safer when you use Python's dedicated path tools instead of manipulating strings by hand. The pathlib module provides a Path object that represents a filesystem path, and it offers methods for joining parts, checking existence, listing contents, reading, and writing. Because Path understands the conventions of the operating system you are on, code written with it tends to work on different systems without change. That portability is a real advantage for scripts that might be shared or moved.

Joining path parts is a good example. Gluing strings together with slashes can produce double slashes, missing slashes, or the wrong separator on another system. Joining Path objects, or using the division operator that pathlib provides, handles these details for you and produces a clean path. The resulting code is also easier to read, because it shows the structure of the path: a base folder, a subfolder, and a filename, each as a separate part, rather than one long string that must be parsed by eye.

Path objects also make common checks simple. You can ask whether a path exists, whether it is a file or a folder, what its name and suffix are, and what its parent folder is. These checks let a script validate its inputs clearly, and produce helpful messages when something is missing. For example, instead of failing with a cryptic error when trying to open a file, a script can first check whether the path exists and report the full path it expected. That small change makes debugging far easier, especially for someone else running your script.

Reading and writing files through Path is similarly convenient. A Path can read text, write text, and create parent folders, all with short, readable calls. For small files, this is often clearer than opening a file, reading it, and closing it manually. For larger files, or for files that need special handling, the traditional open approach remains appropriate, and Path can still supply the path itself. The two styles work well together, and choosing between them is a matter of clarity for the task at hand.

Finally, paths remind you to think about relative and absolute locations. A relative path depends on the current working directory, while an absolute path does not. For scripts run by hand, relative paths are convenient. For scheduled scripts, absolute paths, or paths derived from the script's own location, are more reliable. Whichever you choose, state it clearly in your code or documentation, so that the script's behaviour does not depend on a hidden assumption.

Another advantage is that Path objects can be passed around and combined without losing track of which part is a folder and which part is a filename, which reduces mistakes when a script builds output paths from input paths in a loop.

Check: Why is using pathlib preferable to gluing path strings together by hand?

Section 16

Glob patterns are a concise way to describe a set of files by their names. You have probably met them in a terminal: an asterisk matches any sequence of characters, so *.txt means all files ending in .txt, and a question mark matches a single character. Python supports glob patterns through its path tools, allowing a script to find all files matching a pattern in a folder, or in a folder and its subfolders. This is a powerful way to select input files without listing every name by hand.

The strength of glob patterns is also their risk. A pattern that is too broad can match files you did not intend, including files in subfolders if you use a recursive pattern. A pattern that is too narrow can miss files, producing an incomplete result that looks complete. It is therefore worth testing a pattern before using it in a script that changes files. A small script that simply prints the files matching a pattern lets you see exactly what would be processed, and it is a good companion to the dry-run habit from earlier in this chapter.

Case sensitivity is another detail to watch. On some systems, *.TXT and *.txt match different files, while on others they match the same files. If your data contains a mixture of upper- and lower-case endings, a single pattern may miss some files. A script that needs to handle both can either use two patterns, or filter matched files by checking their endings in a case-insensitive way. Being explicit about this choice avoids a script that works on one computer and quietly misses files on another.

Recursive globs deserve care. A pattern that searches subfolders can be very useful, for example when gathering reports from a tree of project folders. It can also be surprising, because it may reach into backup folders, archive folders, or folders you had forgotten about. Before using a recursive pattern in a script that modifies files, print the full list of matches and check it for unexpected locations. If the list includes folders that should be excluded, refine the pattern or add an explicit filter.

Finally, remember that glob patterns match names, not content. A file named report.txt that actually contains something else will still match *.txt. If the content matters, the script must open and check the file, not rely on its name alone. Glob patterns are a selection tool, and like all selection tools, they work best when combined with validation of what is actually found.

It is also worth remembering that a glob pattern is evaluated at the moment the script runs, so files created after that moment will not be included, and files deleted before it will not appear, even if they existed when you wrote the code.

Check: Why should you print the files matched by a glob pattern before using it to change files?

Section 17

Many scripts need settings: input folders, output locations, thresholds, or names that might change. Hard-coding every setting into the script makes it inflexible, because changing a folder means editing code. Reading settings from a configuration file separates the what from the how: the code describes the task, and the configuration file describes the particular run. That separation makes a script easier to reuse, share, and adjust without touching its logic.

Configuration files come in several common formats. Simple key-and-value files are easy to read and edit by hand. JSON files suit structured settings, such as lists and nested options, and Python can read them with its standard library. TOML is another readable format that has become popular for Python project settings. The best choice depends on who will edit the file. If a non-programmer will edit it, a simple, forgiving format with comments is best. If the settings are structured and machine-generated, JSON or TOML may be clearer. Whichever format you choose, document each setting with a comment or an example file.

A script that reads configuration should handle missing or invalid settings gracefully. If a required setting is absent, the script should stop with a message naming the setting and the file it expected. If an optional setting is absent, the script should use a sensible default and, ideally, mention that it did so in its log. Silently using a default for a setting that the user thought they had set can cause confusion, especially when the default points at a different folder. Clear messages about which settings were used turn a mysterious run into an understandable one.

It is also wise to keep an example configuration file alongside the script. The example shows every setting, with comments explaining what each one does, and sensible example values. A new user can copy the example, change the values they need, and run the script without reading the code to discover the settings. That small act of documentation greatly increases the chance that someone else, or your future self, will use the script correctly.

Finally, be careful about what goes into a configuration file. Passwords, secret keys, and other credentials should not be stored in plain configuration files that might be shared or copied. That topic belongs to the security chapter, but the principle starts here: configuration is for settings that are safe to share, such as folders, thresholds, and names. Secrets need a different, safer home, which you will learn about later.

When a configuration value changes the behaviour of a script in an important way, such as choosing a folder to delete files from, it deserves an extra check, for example by printing the value at the start of the run so that a mistake is visible before any changes are made.

Check: What is the advantage of reading settings from a configuration file instead of hard-coding them?

Section 18

Environment variables are named values stored outside a script, in the environment where the script runs. They are a common way to pass settings, especially settings that differ between computers or that should not be written into code or configuration files. For example, a script might read the folder it should watch, or the name of the user running it, from an environment variable. Python can read these values through its operating-system interface, with a default if a variable is not set.

Environment variables are particularly useful for sensitive settings, because they keep secrets out of files that might be shared. Instead of writing a password into a script, the script reads it from an environment variable that is set only on the computer where it runs. This does not make the secret perfectly safe, since anyone with access to that computer's environment might see it, but it avoids the common and serious mistake of committing a secret into a shared file or a version-control repository. You will learn more about handling secrets safely in the security chapter.

There are also drawbacks. Environment variables are invisible in the script itself, so a person reading the code cannot tell what values it will use without checking the environment. That can make debugging harder: a script that works on one computer may fail on another simply because a variable is missing or set differently. To reduce this problem, a script should check for required variables at the start, report clearly which ones are missing, and use sensible defaults for optional ones. Logging which settings were used, without logging secret values, helps here too.

Another drawback is that environment variables are always text. A script that needs a number, a list, or a true-or-false value must convert the text itself, and handle the case where the text does not convert. For example, a variable set to the word "yes" is not automatically the same as a true value in code. Writing a small, explicit conversion, with a clear error if the value is invalid, prevents subtle bugs where a setting is silently ignored because its text form was not recognised.

Finally, environment variables are best used sparingly. A script that depends on twenty variables is hard to run, because setting up the environment becomes a task in itself. Use them for settings that genuinely vary by environment or that are sensitive, and use configuration files or command-line options for the rest. That balance keeps scripts both flexible and understandable.

On a shared computer, remember that environment variables set for one user are not automatically available to another, so a script that relies on them should document which variables it needs and what values are expected. That documentation saves time for anyone setting the script up for the first time.

Check: Why are environment variables often used for sensitive settings rather than configuration files?

Section 19

Sometimes a script needs to run another program, such as a system tool that converts a file, compresses a folder, or checks a network service. Python's subprocess module lets a script start another program, pass it arguments, and capture its output. This is powerful, because it lets a Python script reuse tools that already exist, rather than reimplementing them. It also requires care, because running another program means trusting that program and its arguments, and handling its failures.

The safest way to run another program is to pass its arguments as a list, with the program name first and each argument as a separate item, rather than as a single string that a shell must interpret. When arguments are passed as a list, a filename containing a space or a special character is treated as one argument, exactly as intended. When a single string is passed to a shell, special characters can change the meaning of the command in surprising and dangerous ways. Avoiding the shell, unless you genuinely need its features, is therefore a strong default.

A script should also check whether the other program succeeded. Every program reports an exit status when it finishes, with zero usually meaning success and other values meaning failure. Python can check this status and raise an error if the program failed, rather than continuing as if everything worked. Capturing the other program's output, and including relevant parts of it in the log or error message, makes failures far easier to diagnose than a bare message that something went wrong.

Timeouts are another useful safeguard. A program that normally finishes in seconds might, on a bad day, hang indefinitely, and a script waiting for it would hang too. Setting a timeout tells Python to stop waiting after a chosen period and report the problem. For scheduled scripts, this is especially important, because a hung script can block later runs. Choosing a generous timeout, one that allows slow but normal runs to finish, while still catching genuine hangs, is a matter of judgement based on the tool involved.

Finally, only run programs you trust, with arguments you have validated. If a filename or other input comes from an untrusted source, check it before passing it to another program, and never let untrusted input become part of a shell command. Subprocess automation is a legitimate and common technique, but it inherits the risks of the programs it runs. Treating those programs as partners that must be checked, not as magic that always works, keeps the script safe.

If the other program produces a large amount of output, capturing it all at once can use a lot of memory, so for long-running tools it may be better to stream the output or redirect it to a file and review it afterwards.

Check: Why should subprocess arguments usually be passed as a list rather than a single shell string?

Section 20

As scripts grow, organising them well becomes important. A single file is fine for a short script, but a script that has grown to several hundred lines, with many functions, becomes hard to navigate. Splitting it into modules, each with a clear responsibility, makes the code easier to read and test. For example, one module might handle reading input files, another might handle transforming data, and a third might handle writing output. Each module can then be understood on its own, and changes in one area are less likely to break another.

A useful organising principle is to separate the core logic from the parts that deal with the outside world. Functions that transform data, such as cleaning a row or calculating a total, are easy to test because they take a value and return a value, without touching files. Functions that read files, write files, or run other programs are harder to test, because they have side effects. Keeping the pure, testable logic separate from the file-handling code means that most of the script can be checked quickly, and only the outer layer needs careful testing with real files.

Naming is part of organisation too. A script file named script.py tells a future reader nothing, while a name such as summarise_sales.py says what the script does. Function names should similarly describe their job, not their implementation. A function called clean_row is clearer than one called process, and a variable called input_folder is clearer than one called x. Good names reduce the need for comments, because the code explains itself, and they make errors easier to find, because a reader can guess what each part is supposed to do.

It is also helpful to include a short description at the top of a script, explaining what it does, what input it expects, and what output it produces. This header need not be long. A few lines, plus an example of how to run the script, can save a future user, including yourself in six months, a great deal of puzzling. If the script takes options or reads a configuration file, the header is a natural place to mention them, even if fuller documentation lives elsewhere.

Finally, keep a script's dependencies visible. If it uses only the standard library, say so. If it needs an extra package, mention that, and ideally list it in a requirements file. A script that works on your computer because a package happens to be installed, but fails on another computer where it is not, is a common frustration. Making dependencies explicit turns a mysterious failure into a simple installation step.

A small README file in the same folder as a script can also help, by recording its purpose, its inputs, its outputs, and an example command, in a place where anyone finding the script will naturally look first.

Check: Why is it helpful to separate pure data-transforming functions from file-handling code?

Section 21

Reusable functions are the building blocks of good automation. When you notice that the same few lines appear in several places, or in several scripts, turning them into a function saves effort and reduces mistakes. Instead of copying and pasting code, with the risk that one copy is later fixed and another is not, you write the logic once, give it a clear name, and call it wherever it is needed. If the logic needs fixing, you fix it in one place, and every caller benefits. That single change is the heart of reuse.

A reusable function should do one job, and its name should say what that job is. A function that reads a CSV file, cleans its rows, calculates totals, and writes a report is doing too much, and it will be hard to reuse for a different report. Smaller functions, such as one that cleans a single row and one that calculates a total from clean rows, can be combined in different ways. Thinking about the smallest useful job a function can do is a skill that improves with practice, and it pays off every time you reuse the function.

Inputs and outputs should be explicit. A function that reads from a global variable, or that quietly changes a file as a side effect, is harder to reuse than one that takes its input as an argument and returns its result. The caller can then decide what to do with the result, whether to print it, write it to a file, or pass it to another function. This style also makes testing easier, because you can call the function with sample data and check its return value, without setting up files or worrying about side effects.

Documentation matters for reusable functions, even within a single script. A short docstring explaining what the function does, what arguments it expects, and what it returns helps a future reader, including yourself, use it correctly. It also forces you to think clearly about the function's contract: if you cannot describe what it does in a sentence or two, it may be doing too much, or its purpose may be unclear. That moment of clarification often leads to a better design.

Finally, reuse can extend beyond one script. Functions that prove useful in several scripts can be moved into a shared module, which other scripts import. That step introduces questions about where the module lives and how it is found, but the principle remains the same: write once, use many times, and fix once. Starting with reuse inside a single script builds the habits that make sharing across scripts natural later.

When a function is reused in several scripts, changing its behaviour can affect all of them, so it is worth noting its purpose and its expected inputs clearly, and testing callers after any change.

Check: Why is copying and pasting code worse than calling a reusable function?

Section 22

Testing a script manually is not a lesser form of testing; for small automation, it is often the main form, and it deserves to be done systematically. The idea is simple: run the script on sample data where you already know the correct answer, and compare the script's output with that answer. If they match, you gain confidence. If they do not, you have a concrete case to investigate, which is far more useful than a vague feeling that something might be wrong.

Good sample data is deliberately varied. It should include ordinary cases, edge cases, and awkward cases. For a renaming script, that means files with spaces, files with unusual characters, files that are already correctly named, and files that would collide with each other. For a summarising script, it means empty files, files with missing values, and files with values at the extremes of the expected range. Creating such a test folder takes a little time, but it catches problems that a single happy-path example would miss entirely.

It helps to keep the test folder separate from real data, and to rebuild it before each test run, so that a previous run's output does not contaminate the next test. A script that appends to output, for example, will produce different results on a second run if the output file is not cleared first. Rebuilding the test data from scratch, or using a fresh output folder each time, removes that source of confusion. The goal is that a test run is repeatable: the same input should always produce the same output.

Manual testing also includes reading the output, not just checking that the script finished without an error. A script can finish successfully and still produce wrong output, if its rule is wrong or its filter is too broad. Reading a sample of the output, checking counts against expectations, and opening a result in the program that will ultimately use it, such as a spreadsheet, are all part of testing. A script that runs without crashing has passed only the lowest bar.

Finally, record what you tested. A short note, even in a comment or a test folder readme, listing the cases you tried and the results you expected, helps you retest after a change. When you later modify the script, you can rerun the same cases and confirm that nothing that worked before has broken. That habit, applied consistently, is the foundation of the more formal testing you will meet in later chapters, and it is already valuable on its own.

It is also useful to test what happens when the script is run twice in a row on the same sample data, because a script that works once but duplicates output on a second run will cause confusion in real use.

Check: Why should test data include awkward and edge cases, not just ordinary ones?

Section 23

Handling large files requires a different mindset from handling small ones. When a file fits comfortably in memory, reading it all at once is simple and fast. When a file is very large, reading it all at once can use so much memory that the script slows down or fails. The solution is usually to process the file in a stream: read a little at a time, process that piece, write any output, and then move on, discarding the piece you have finished with. That way, the script's memory use stays roughly constant, no matter how large the file grows.

Line-by-line processing is the most common streaming pattern for text files. Each line is read, examined, and either kept, transformed, or counted, before the next line is read. This pattern suits logs, CSV files, and many kinds of exported data, where each line is a self-contained record. It also makes progress visible: a script can report how many lines it has processed, which is reassuring for a long run, and it can be stopped and restarted more easily than a script that must load everything before it starts.

There are cases where line-by-line processing is not enough, because a record spans several lines, or because a calculation needs the whole file, such as finding a median value. In those cases, you may need to make two passes over the file, first gathering the information needed, then producing the output, or you may need to store a compact summary rather than the full data. For example, to count how many times each word appears, you need to store counts, not every word. Choosing a compact representation is often the key to handling large data with modest memory.

Chunking is another useful technique. Instead of reading one line at a time, a script reads a block of lines, processes the block, and then moves to the next block. Chunking can be faster than line-by-line processing for some tasks, while still keeping memory use bounded. The right chunk size depends on the task and the machine, but the principle is the same: never hold more than you need at any one moment. If you are unsure, start with line-by-line processing, which is simplest, and move to chunking only if speed becomes a problem.

Finally, test large-file scripts on a large test file, not just a small sample. A script that is fast on a thousand lines may be slow on a million, if it does something expensive for every line, such as searching a growing list. Generating a large test file, by repeating sample data many times, lets you measure the script's behaviour before it meets real data. That measurement turns a guess about performance into evidence.

Check: Why does streaming a large file keep memory use roughly constant?

Section 24

Data cleaning is the unglamorous but essential work of turning messy real-world data into a form that can be used reliably. Real data often contains inconsistent spellings, mixed date formats, stray spaces, empty cells, and duplicate rows. A script that summarises such data without cleaning it first will produce results that are technically correct for the data as given, but misleading for the question being asked. For example, if the same place appears under two spellings, a count by place will split its total in two, and nobody will notice unless they look closely.

The first step in cleaning is to understand the mess. Before writing code to fix anything, examine a sample of the data and list the kinds of problems you find. Are dates written in several formats? Are some numbers stored as text with currency symbols? Are there blank lines, or lines with missing fields? That inventory guides the cleaning rules, and it prevents you from writing fixes for problems that do not exist, while missing the ones that do. It also gives you a way to measure progress: after cleaning, the same sample should show fewer problems.

Cleaning rules should be conservative. It is tempting to delete anything that looks odd, but odd data is sometimes correct data. A very large value might be a typo, or it might be a genuine outlier that matters. A good cleaning script therefore distinguishes between problems it can fix safely, such as trimming stray spaces or standardising capitalisation, and problems it should flag for a person, such as values that are wildly out of range. Flagging, rather than silently deleting, preserves information and lets a human decide.

It is also important to keep a record of what was cleaned. A cleaning script should report how many rows it changed, how many it flagged, and, ideally, write flagged rows to a separate file for review. That report lets you judge whether the cleaning behaved as expected, and it gives you a trail back to the original data if a cleaning rule proves wrong. Without such a report, cleaning becomes an invisible step, and errors introduced by cleaning are hard to trace.

Finally, clean data should be written to a new file, leaving the original untouched, as discussed earlier in this chapter. That way, if a cleaning rule turns out to be mistaken, you can revise the rule and rerun the script on the original data, rather than trying to undo changes to an already cleaned file. Data cleaning is iterative: you clean, you check, you refine, and you clean again. Keeping the original safe makes that iteration possible.

After cleaning, comparing simple counts before and after, such as the number of rows and the number of distinct values in a key column, can reveal whether the cleaning changed more than you intended.

Check: Why should a cleaning script flag unusual values for review instead of deleting them silently?

Section 25

Generating reports is a common goal of automation: turning raw data into a summary that a person can read and act on. A good report answers a question. It might show total sales by week, the number of errors by type, or the most common problems in a set of records. The first step in building a report script is therefore to state the question clearly. A script built around a vague goal, such as summarising the data, tends to produce a report that is hard to interpret, because neither the writer nor the reader knows what it is supposed to show.

Once the question is clear, the next step is to decide on the report's structure. A report usually has a title, a date or period it covers, a summary section, and then details. Tables are useful for precise values, while short paragraphs can explain what the numbers mean. A script that produces a report should produce the same structure every time, so that readers learn where to look for the information they need. Consistency matters more than decoration: a plain, predictable report that arrives on time is more useful than a beautiful report that changes layout every week.

Calculations in a report script deserve care, especially around totals and averages. Totals should be checked against the raw data, at least on a sample, to confirm that no rows were missed or counted twice. Averages should be accompanied by counts, because an average over three items means something different from an average over three thousand. If a report includes percentages, the denominator should be clear, or the reader cannot judge what the percentage means. These details are what make a report trustworthy, rather than merely plausible.

Output format is another decision. A report might be a plain text file, a CSV file for further analysis, or a simple HTML page that can be opened in a browser. Each format has strengths. Plain text is simple and portable. CSV is useful when the reader wants to sort or chart the data. HTML allows headings and tables that are pleasant to read. The choice depends on who will read the report and what they will do with it, and a script can produce more than one format from the same calculations if needed.

Finally, a report script should record its own metadata: when it ran, what input it used, and how many records it processed. That information, placed at the top or bottom of the report, lets a reader judge whether the report is current and complete. A report without a date is a trap, because it may be read weeks later as if it were fresh. A dated, counted, and clearly structured report is a reliable product of automation.

Check: Why should a report include the count behind an average?

Section 26

Notifications are how an automated script tells a person that something needs attention, or that a job has finished. A script that runs unattended and finds a problem, but tells nobody, has only done half its job. The other half is communication. Notifications can take many forms: a message in a log file, an email, a message in a team chat, or a simple summary printed when the script is run by hand. The right form depends on how urgent the message is, and on how the recipient prefers to receive it.

It is important to distinguish between routine reports and alerts. A routine report, such as a daily summary, is expected, and the recipient will look for it. An alert, such as a message that a backup failed, is unexpected, and it should stand out. If a script sends an alert for every minor event, recipients will learn to ignore its messages, and a genuine alert will be missed. This is often called alert fatigue, and it is a real risk in automation. Designing notifications so that routine messages are quiet and alerts are rare and clear is therefore a matter of safety, not just politeness.

A notification should contain enough information to act on, without requiring the recipient to hunt for details. At minimum, it should say what happened, when it happened, which script or system was involved, and what the recipient should do next, if anything. A message that says only that something failed, without naming the script, the input, or the error, forces the recipient to investigate from scratch. A message that includes those details lets them decide quickly whether the problem is urgent, and where to start fixing it.

Timing matters too. A script that checks for new files every hour should not send a message every hour saying that nothing new arrived, unless the recipient has asked for that reassurance. More often, silence means success, and a message means something changed or failed. That convention, sometimes called no news is good news, keeps the volume of messages low and makes each message meaningful. For scheduled reports, the opposite convention applies: the report itself is the message, and its absence is the alert.

Finally, test notifications before relying on them. Send a test message through the same route the script will use, and confirm that it arrives, that it is readable, and that it contains the right details. A notification system that has never been tested is a hope, not a safeguard. Testing it once, and retesting after any change to the script or the route, turns that hope into a reliable link between the script and the person responsible for it.

If a notification route changes, for example because a team moves to a different chat tool, update the script and test the new route immediately, rather than discovering the break during a real incident.

Check: What is alert fatigue, and why should routine messages stay quiet?

Section 27

Gathering data from the web automatically is a powerful technique, but it carries responsibilities that do not apply to processing files on your own computer. Websites are run by other people, and their content, their servers, and their terms of use deserve respect. Before writing a script that fetches pages automatically, check whether the site offers an official way to get the data, such as a feed or an application programming interface. Using an official route, where one exists, is usually more reliable, more polite, and less likely to break when the site changes its layout.

If no official route exists, and you still need to fetch pages, do so gently. A script that requests pages as fast as possible can overload a small site, effectively denying service to other visitors. Adding a pause between requests, fetching only the pages you need, and avoiding repeated fetching of the same page are basic courtesies. They also make your script more robust, because a site that is not overloaded is less likely to block your requests. Think of it as visiting a shop: you would not run through the aisles grabbing everything at top speed, and the same restraint applies online.

Respect for robots files and terms of use is part of this. Many sites publish a file that tells automated visitors which parts of the site they may fetch. While the details of that file are beyond this chapter, the principle is clear: if a site asks automated visitors not to fetch a section, honour that request. Similarly, if a site's terms forbid automated access, do not circumvent them with tricks. There are usually legitimate alternatives, such as asking permission, using a public dataset, or gathering the data by hand for a small sample.

Copyright and privacy matter too. Just because data is visible on a public page does not mean you may republish it, or use it for any purpose. Facts may be usable in your own summary, but copying large amounts of text, images, or personal information without permission can infringe rights or harm people. When in doubt, use web data for private analysis, summarise it in your own words, and do not republish personal details. That caution protects both the people whose data it is and you as the script's author.

Finally, web automation is fragile. Sites change their layout, their structure, and their protections, often without warning. A script that worked yesterday may fail today, and that failure should be handled gracefully, with a clear log message, rather than by retrying aggressively. Building web gathering scripts with gentle pacing, clear logging, and a willingness to stop when asked is the responsible way to use this power.

Keeping a record of which pages were fetched, and when, also helps you avoid fetching the same data repeatedly, and makes it easier to explain your work if a site owner asks about it.

Check: Why should a web-gathering script pause between requests instead of fetching as fast as possible?

Section 28

Converting files from one format to another is a frequent automation task. A set of images might need to be resized, a document might need to become plain text, or a spreadsheet might need to become CSV for further processing. The pattern is familiar: read the input, transform it, and write the output in the new format. What makes conversion scripts interesting is the attention to quality and to loss. Some conversions are lossless, meaning nothing is lost. Others lose information, such as formatting, colour depth, or precision, and that loss should be understood before the conversion runs.

The first step is to identify the right tool. Python's standard library handles many text conversions well, and well-known third-party tools handle images, documents, and other formats. Using an established tool, rather than writing your own converter from scratch, is usually wiser, because format details are full of edge cases that established tools have already handled. The script's job is then to orchestrate: find the files, call the tool with the right settings, check that it succeeded, and organise the output. That orchestration is where automation adds value.

Quality settings deserve explicit choices. For images, that might mean choosing a size and a quality level. For text, it might mean choosing an encoding. For numbers, it might mean choosing how many decimal places to keep. A script that accepts default settings without thinking about them may produce output that is too large, too blurry, or too imprecise for its purpose. Stating the settings in the script, or in its configuration, makes the conversion repeatable and lets a reader judge whether the output is fit for its intended use.

As with other file-changing automation, conversion scripts should keep the original files untouched until the output has been checked. Converting in place, where the original is replaced immediately, leaves no way back if the conversion proves wrong. Writing to a new folder, checking a sample of the output, and only then deciding whether to replace the originals is safer. A log that records how many files were converted, how many failed, and which settings were used completes the picture.

Finally, think about naming and organisation of the output. Converted files should have clear names, preferably derived from the original names with a changed suffix, and they should live in a folder that makes their relationship to the originals obvious. A conversion that scatters files with confusing names across a folder creates a new mess while solving an old one. Clean output organisation is part of the conversion's success, not an afterthought.

When converting many files, processing them in a clear order, such as alphabetical order, makes progress easy to follow and makes it obvious if the script stops partway through the set.

Check: Why should a conversion script keep original files untouched until output is checked?

Section 29

Archiving is the practice of moving files that are no longer actively needed into long-term storage, in an organised way, so that they can still be found if needed. Without archiving, working folders fill up with old files, making it harder to find current work and increasing the risk that an old file is mistaken for a current one. A script can automate archiving by identifying files that have not been changed for a chosen period, or that match a pattern for completed work, and moving them into an archive folder organised by date or by project.

The rule for what counts as archivable deserves careful thought. A rule based purely on age, such as moving any file untouched for a year, might archive a file that is still needed occasionally. A rule based on location, such as archiving everything in a folder marked completed, is clearer, because a person has already decided that the work in that folder is finished. Combining rules, for example archiving completed folders after a waiting period, gives both human judgement and automatic tidiness. Whatever the rule, it should be stated clearly in the script and in its documentation, so that nobody is surprised by a move.

Safety is paramount in archiving, because moving files feels less dramatic than deleting them, but a move to the wrong place can be just as confusing. A dry run, showing which files would be moved and where they would go, is an excellent first step. A log of actual moves, mapping old locations to new ones, provides a way back if a move proves wrong. And, as always, testing on a copy of the data before running on real folders turns a potentially disruptive operation into a routine one.

Archiving and deleting are different, and they should not be confused. Archiving preserves files, just in a different place. Deleting removes them, and should be a separate, deliberate decision, with its own safeguards. A script that archives should never delete as a side effect, unless that behaviour is clearly documented and explicitly intended. Keeping the two operations separate prevents a tidying script from becoming a data-loss script by accident.

Finally, an archive is only useful if it can be searched. Organising archived files by year, by project, or by another clear scheme, and keeping a simple index or log of what was archived and when, makes retrieval possible. An archive that is just a pile of files in a forgotten folder is little better than a cluttered working folder. Thoughtful organisation turns archiving from mere tidiness into a reliable record.

Before archiving, it is worth checking that the archive destination has enough free space, because a move that fails halfway can leave files split between two locations and create confusion.

Check: How does archiving differ from deleting, and why should the two stay separate?

Section 30

Scripts, like any code, need maintenance. A script that worked perfectly when it was written may fail a year later, because its input format changed, a folder moved, or a tool it relies on was updated. Maintenance is the work of keeping a script working as its surroundings change, and it is easier if the script was written with maintenance in mind. Clear names, short functions, a header describing its purpose, and a log of what it does all make maintenance easier, because they help a future reader, who may be you, understand the script quickly.

One useful maintenance habit is to review scripts periodically, especially those that run on a schedule. A scheduled script that fails quietly can go unnoticed for weeks, as discussed earlier, so checking its logs on a regular schedule is part of maintenance, not an optional extra. When you review a script, ask whether its inputs still look the same, whether its output is still used, and whether its purpose is still relevant. A script whose purpose has disappeared should be retired, not left running out of habit, because a running script that nobody needs is a source of confusion and potential errors.

Changes to a script should be made carefully, and tested on sample data before being applied to real data. This is the same discipline as writing the script in the first place, applied to its later life. If a script is important, keeping it under version control, so that previous versions can be recovered, is wise. Version control also records why changes were made, if commit messages are written clearly, which turns the script's history into a story that a maintainer can follow.

Documentation should be updated when a script changes. If a script gains a new option, or its input folder changes, its header, its example configuration, and any notes about it should be updated at the same time. Documentation that lags behind the code is worse than no documentation, because it actively misleads. Updating documentation as part of the change, rather than as a separate task to be done later, keeps the two in step.

Finally, know when to retire or replace a script. A script that has grown so complex that it is hard to maintain may have outgrown scripting, and may need to be rebuilt as a more structured program, with tests and a proper design. Recognising that moment is not a failure; it is a sign that the automation proved valuable enough to deserve a larger investment. The skills in this chapter, planning, testing, logging, and careful file handling, carry over to that larger program, and they will serve you well there too.

A short changelog, even as comments at the top of a script, can also help a maintainer see at a glance what changed, when, and why, without digging through older versions.

Check: Why should a scheduled script's logs be checked regularly, even when nothing seems wrong?

Section 31

Operating systems provide schedulers that run programs at chosen times, and understanding their basic ideas helps you use them well, whatever system you are on. On Unix-like systems, a scheduler reads a table of jobs, each with a schedule expressed in minutes, hours, days, and months, and runs the listed command when the schedule matches. On Windows, a task scheduler offers a graphical way to set up similar jobs, with triggers, conditions, and actions. The details differ, but the questions you must answer are the same: what should run, when, as which user, and where should its output go.

When you set up a scheduled job, be explicit about paths. A job run by a scheduler often starts in a different working directory from the one you use in a terminal, and it may have a different environment, with fewer variables set and a different search path for programs. A script that relies on relative paths, or on a program found only because your terminal sets up a particular path, may fail when scheduled, even though it works by hand. Using absolute paths for the script, its inputs, and its outputs, and setting any needed environment variables in the job itself, removes a whole class of mysterious failures.

Permissions matter for scheduled jobs too. A job runs as a particular user, and it can only read files that user can read and write files that user can write. A script that works when you run it may fail when scheduled under a different account, because that account lacks access to a folder. Testing a scheduled job under the same account and conditions it will use in production, rather than assuming it will behave the same, catches these problems early. If a job needs access to a protected folder, granting that access deliberately, rather than running the job with excessive privileges, is the safer choice.

Output handling is another practical detail. A scheduled job's screen output usually goes nowhere unless you direct it to a file, so directing both normal output and error output to a log file is essential. That log is how you know the job ran, what it did, and whether it failed. Some schedulers also offer to send output by email, but a log file is a simpler and more reliable record, because it can be checked at any time, not just when a message arrives.

Finally, keep a list of your scheduled jobs, with their purpose and schedule, somewhere you can find it. It is easy to forget about a job set up months ago, especially if it runs quietly and successfully. A forgotten job that later starts failing, or that conflicts with a new job, can cause confusion. A simple document listing each job, what it does, and when it runs turns scheduling from a set of hidden habits into a system you can manage.

Check: Why might a script work by hand but fail when run by a scheduler?

Section 32

Maintainability is the quality that determines whether a script remains useful over time, or becomes a puzzle that nobody wants to touch. A maintainable script is easy to read, easy to change, and easy to test. Those qualities do not happen by accident; they come from choices made while writing, such as using clear names, keeping functions short, and avoiding clever tricks that save a line but cost an hour of puzzling later. When you write a script, imagine that the person maintaining it is you in a year's time, with no memory of why you wrote it that way.

Readability is the foundation of maintainability. Code is read far more often than it is written, so optimising for the reader is wise. Short functions with clear names, consistent formatting, and comments that explain why, rather than restating what the code obviously does, all help a reader. A comment that says why a particular threshold was chosen, or why files are processed in a certain order, is valuable, because that reasoning is not visible in the code itself. A comment that merely repeats the code adds noise and should be avoided.

Avoiding duplication is another maintainability habit. When the same logic appears twice, a future change must be made twice, and it is easy to miss one copy. Extracting the shared logic into a function, as discussed earlier, solves this. The same principle applies to values: a folder path or a threshold that appears in several places should be defined once, at the top of the script or in a configuration file, and referred to by name elsewhere. Changing it then requires one edit, not a search through the whole file.

Simplicity also matters. A script that uses the simplest tool that does the job is easier to maintain than one that uses an elaborate technique for a simple task. Elaborate techniques may be impressive, but they require more understanding to change safely. When a simpler approach works, choose it, and reserve complex techniques for problems that genuinely need them. That restraint keeps scripts approachable, especially for colleagues or future maintainers who may be less experienced.

Finally, maintainability includes knowing when to stop adding features. A script that does one job well is easier to maintain than a script that has grown to do five jobs, each with its own options and exceptions. When a new requirement arrives, ask whether it belongs in this script, or whether it deserves its own script. Keeping scripts focused is a form of discipline, and it pays off every time the script needs to be understood or changed.

A maintainable script also benefits from a consistent style, because a reader who knows where to look for settings, where the main logic lives, and how errors are reported can understand a new script far more quickly than one organised differently each time.

Check: Why should comments explain why a choice was made, rather than restating what the code does?

Section 33

Version control is a system for tracking changes to files over time, and it is valuable for scripts even when only one person works on them. The most widely used system is Git, and its basic idea is simple: you record snapshots of your work, called commits, each with a message describing what changed and why. Later, you can look back at those snapshots, compare them, and restore an earlier version if a change proves wrong. That safety net alone makes version control worth learning for anyone who writes scripts they care about.

For scripts, version control solves several practical problems. It provides a backup of your work, in the sense that committed snapshots are preserved even if you accidentally delete or overwrite a file. It records the history of a script, so you can see when a particular feature was added or a bug was fixed. And it makes experimentation safer, because you can try a change, and if it does not work, return to the last good snapshot without manually undoing your edits. That freedom to experiment encourages improvement, because the cost of a failed experiment becomes very low.

Good commit messages are part of the value. A message such as "fixed bug" tells a future reader almost nothing, while a message such as "skip empty CSV rows instead of counting them as errors" explains what changed and why it mattered. Writing such messages takes a few seconds, but it turns the history into a useful record rather than a list of vague entries. When you later wonder why a script behaves a certain way, the commit history, read alongside the code, often provides the answer.

Version control also helps when scripts are shared. If several people work on the same script, version control tracks who changed what, and it helps combine changes without overwriting each other's work. Even for a single author, sharing a script through a version-controlled repository makes it easier for others to obtain the latest version, report problems, and suggest improvements. The repository becomes the script's home, rather than a copy passed around by email or chat, which quickly becomes outdated.

Finally, version control is not a substitute for backups or for testing. A repository on the same computer does not protect against losing that computer, so an additional copy elsewhere is still wise for important work. And a committed script is not necessarily a correct script; it still needs testing. What version control provides is a reliable history and a safe way to experiment, and those are valuable additions to the habits you have already learned in this chapter.

If you work on a script on more than one computer, version control also makes it easier to move work between them, because you can send committed snapshots from one machine to another instead of copying files by hand and wondering which copy is newest.

Check: How does version control make experimenting with a script safer?

Section 34

Documentation for a script answers three questions for its reader: what does it do, how do I run it, and what should I expect? A script without documentation may be perfectly clear to its author on the day it is written, but it becomes mysterious surprisingly quickly, even to that same author. Writing documentation is therefore not a courtesy to strangers; it is a gift to your future self, and a practical part of writing a script that will actually be used.

The simplest useful documentation is a header at the top of the script file. It should state the script's purpose in a sentence or two, list its inputs and outputs, give an example command for running it, and mention any configuration or environment variables it needs. That header travels with the script wherever it goes, so it is never lost, unlike a separate note that might be filed elsewhere. Keeping the header short increases the chance that it will be read and kept up to date.

For scripts that are used by others, a separate readme file is also valuable. The readme can explain the script's purpose in more detail, describe its options, show examples of its output, and list its requirements. It can also record known limitations, such as file types it does not handle, or conditions under which it should not be used. Stating limitations honestly prevents misuse, and it helps users decide whether the script is suitable for their task before they run it.

Examples are particularly powerful in documentation. A single example command, with sample input and sample output, often explains a script more clearly than several paragraphs of description. Examples also serve as a form of testing: if you run the example and the output does not match the documentation, either the script or the documentation has changed, and one of them needs updating. Keeping examples accurate is therefore part of maintaining both the script and its documentation.

Finally, documentation should be updated as part of changing a script, not afterwards. When you add an option, change an input format, or alter the output, update the header and the readme in the same change. That discipline prevents the common and frustrating situation where documentation describes a script that no longer exists. A script whose documentation matches its behaviour is a script that others can trust and use with confidence.

Good documentation also helps when a script is handed to someone else, because it answers their first questions without requiring them to interrupt the author, and it reduces the chance that they will use the script in a way it was not designed for.

Keeping documentation close to the script, in the same folder, also makes it easier to find when it is needed most.

Check: What three questions should script documentation answer for its reader?

Section 35

Performance matters for automation, but it matters in a particular way. Most scripts process a modest amount of data, and for those scripts, clarity matters more than speed. A script that takes a few seconds instead of a few tenths of a second is still perfectly usable, if it is correct and easy to understand. Optimising such a script for speed, at the cost of clarity, is a poor trade. The right question is not whether a script is as fast as possible, but whether it is fast enough for its purpose, and whether its speed will remain acceptable as its data grows.

When a script is too slow, the first step is to find out where the time goes, rather than guessing. A script might spend most of its time reading files, waiting for another program, or performing a particular calculation. Each of those causes has a different remedy, and guessing wrong wastes effort. Simple timing, such as recording how long each major step takes, often reveals the bottleneck quickly. That measurement, even if rough, turns optimisation from a guess into a targeted fix, and it prevents you from optimising a step that was never the problem.

Common performance pitfalls in scripts are easy to recognise once you know them. Repeatedly searching a long list, instead of using a faster lookup structure, can make a script slow as data grows. Reading the same file many times, instead of reading it once and reusing its contents, wastes time. Performing an expensive operation inside a loop, when it could be done once before the loop, is another frequent waste. None of these require advanced techniques to fix; they require noticing the pattern and restructuring the code slightly, which is why measuring first is so valuable.

It is also worth remembering that the slowest part of many automation tasks is not the Python code at all, but waiting for disks, networks, or other programs. In those cases, making the Python code faster will have little effect, because the script spends most of its time waiting. Reducing the amount of data read, avoiding unnecessary work, or processing files in parallel, where appropriate, may help more than micro-optimising code. Understanding where time is actually spent keeps optimisation effort focused on what matters.

Finally, keep performance in proportion. A script that runs once a day and takes a minute is fine, even if it could take ten seconds with effort. A script that runs every minute and takes fifty seconds is a problem, because it barely finishes before the next run starts. Judging performance by the script's schedule and purpose, rather than by an abstract desire for speed, leads to sensible decisions and avoids wasted work.

Check: Why should you measure where a script spends time before trying to make it faster?

Section 36

Security in automation starts with a simple principle: a script should only do what it is supposed to do, with only the access it needs, and it should treat input with caution. Scripts often run with the permissions of the person who runs them, which means a script that is tricked into doing something unintended can do real damage, such as overwriting files or revealing information. Thinking about security while writing a script is therefore not an extra for specialists; it is part of writing a trustworthy script.

One of the most important security habits is to keep secrets out of scripts. Passwords, tokens, and private keys should never be written into code, especially code that might be shared or stored in version control. Reading secrets from environment variables or from a protected file, with permissions that limit who can read it, is safer. Even then, a script should avoid printing secrets in its logs or error messages, because logs are often less protected than the script itself. A log that records that a connection failed is useful; a log that records the password used is a leak.

Input validation is another key habit. A script that processes filenames, for example, should check that they are what it expects, rather than trusting them blindly. A filename containing a path that escapes the intended folder could cause a script to read or overwrite files elsewhere, if the script simply joins paths without checking. Validating that a resolved path stays within the intended folder, and rejecting inputs that do not, is a simple guard against a whole class of problems. Similar care applies to any input that will be passed to another program or used to build a command.

Permissions and least privilege complete the picture. A script should run with the minimum permissions needed for its task, not with administrator or root privileges unless those are genuinely required. A script that only reads files and writes a report does not need permission to change system settings, and running it with such permission only increases the damage if something goes wrong. When a script must perform a privileged action, that action should be narrow, deliberate, and clearly documented, not a blanket grant of power.

Finally, remember that security is also about the script's own safety. A script that deletes files should have dry-run mode, backups, and collision checks, as discussed throughout this chapter. Those safety features are security features too, because they limit the damage from both mistakes and malicious input. A script built with caution, validation, and least privilege is a script that can be trusted with real data.

Another simple safeguard is to avoid running scripts downloaded from untrusted sources without reading them first, because a script can do anything its permissions allow, and a brief review often reveals unexpected actions such as sending data elsewhere.

Check: Why should a script never print a secret in its logs or error messages?

Section 37

Cross-platform concerns arise because scripts may run on different operating systems, and those systems differ in small but important ways. Path separators are the most famous example: one system uses a forward slash, another uses a backslash. Line endings differ too, as do the ways that file permissions, case sensitivity, and executable programs are handled. A script that assumes one system's conventions may work perfectly there and fail, or behave strangely, elsewhere. Writing with portability in mind avoids those surprises.

Python's standard library helps a great deal with portability, if you use it. Path objects handle separators for you, text file handling smooths over line ending differences, and operating-system functions provide portable ways to do common tasks. The key is to use those portable tools, rather than relying on system-specific commands or string tricks that assume a particular system. A script that calls a system-specific tool directly will only work where that tool exists, while a script that uses Python's portable facilities will work more widely.

Case sensitivity is a subtle cross-platform issue. On some systems, a file named Report.txt and a file named report.txt are different files; on others, they are the same file. A script that creates both, or that assumes a particular case, may behave differently on different systems. Similarly, a script that searches for files by extension should consider whether case matters for its data, and handle both cases explicitly if needed. Being aware of this difference prevents a script that works in testing from failing, or worse, overwriting a file, on another system.

Another difference is how programs are found and run. A script that runs another program by name assumes that program is installed and available in the search path on every system where the script runs. That assumption often fails. Checking whether the program exists, and reporting a clear error if it does not, is better than failing with a confusing message. Where possible, preferring a Python solution, or documenting the external program as a requirement, makes the script's dependencies clear.

Finally, test on more than one system if the script will be shared. Even if you cannot test on every system, testing on a second system often reveals assumptions you did not know you had made. Portability is not about supporting every system perfectly; it is about avoiding unnecessary dependence on one system's quirks, so that a useful script remains useful wherever it is needed.

File permissions also differ across systems, so a script that relies on a file being executable, or on a particular permission setting, should check and report permission problems clearly rather than assuming the setting it expects is present.

Testing a script with filenames that contain spaces is another simple cross-platform check, because quoting and handling of spaces differ between systems.

Check: Why can a script that assumes one operating system's path separator fail elsewhere?

Section 38

Exit codes are a small but important part of how scripts communicate with the systems that run them. When a program finishes, it reports a number, called its exit code, to the system that started it. By convention, zero means success, and any other value means that something went wrong. Schedulers, other scripts, and command-line tools check this code to decide whether a job succeeded, whether to run the next step, or whether to alert someone. A script that always exits with zero, even when it fails, misleads those systems.

Python scripts exit with zero by default when they finish without an error, and with a non-zero code when an unhandled exception stops them. That default behaviour is often sufficient, but there are cases where explicit exit codes are better. For example, a script that processes many files and skips some might exit with zero if all important work succeeded, or with a distinct code if a required input was missing. Choosing exit codes deliberately, and documenting what they mean, turns a script from a black box into a component that other tools can rely on.

It is also important that a script's exit code matches its log. A script that logs an error but exits with zero sends contradictory signals: a person reading the log sees a failure, while an automated system checking the exit code sees success. Conversely, a script that exits with an error code but logs nothing leaves a person with no clue about what went wrong. Keeping the log and the exit code consistent, so that both tell the same story, is a matter of honesty in automation.

For scripts that are part of a larger workflow, exit codes enable chaining. A workflow might run one script to gather data, then another to summarise it, but only if the first succeeded. That conditional chaining, based on exit codes, prevents a summarising script from running on incomplete or missing data and producing a misleading report. It is a simple mechanism, but it is the foundation of reliable multi-step automation, and it depends entirely on scripts reporting their success or failure truthfully.

Finally, avoid inventing elaborate exit-code schemes unless they are needed. A few well-chosen codes, such as zero for success, one for a general error, and perhaps a specific code for a missing input, are usually enough. Document them in the script's header, use them consistently, and remember that the most important distinction is between success and failure. Everything else is detail.

When a script is run by hand, its exit code is less visible than its printed output, so printing a short success or failure summary at the end helps a human reader see the same result that an automated system would read from the exit code.

Check: Why is it misleading for a script to log an error but exit with code zero?

Section 39

Some scripts ask the user questions while they run, and others run without any interaction at all. Both styles have their place, and choosing between them depends on how the script will be used. An interactive script, which prompts for a filename or a choice, is convenient when a person is present and the choices vary each time. An unattended script, which takes all its settings from arguments, configuration, or environment, is essential when nobody is present, such as when a scheduler runs it. Mixing the two styles carelessly causes problems.

The main risk of interaction in an unattended script is that it waits forever for an answer that will never come. A script that prompts for confirmation before deleting files, for example, will hang when run by a scheduler, because there is no one to answer. That hang can block later runs, fill logs with nothing, and leave work undone. If a script might ever run unattended, it should have a non-interactive mode, where confirmations are either unnecessary, because the script is safe by design, or supplied in advance through arguments or configuration.

Interactive prompts, when used, should be clear and safe. A prompt should say what it is asking, what the default is if the user simply presses enter, and what will happen if they confirm. A prompt that asks "Are you sure?" without saying what is about to happen is not a real safeguard, because the user cannot judge the risk. Similarly, a prompt that requires typing a full word, such as "DELETE", for a destructive action is safer than one that accepts a single keypress, because it forces a moment of attention. Designing prompts with care respects the user's attention and reduces accidental confirmations.

It is also helpful to separate the interactive layer from the core logic. If the core logic takes its settings as arguments, then an interactive script can gather those settings by prompting, while an unattended script can supply them directly, and both can share the same core. That separation avoids duplicating logic, and it makes testing easier, because the core can be tested without simulating user input. It is the same separation principle met earlier in this chapter, applied to user interaction.

Finally, when a script does interact, it should handle unexpected input gracefully. A user might type something the script did not expect, or press the interrupt key to stop it. Handling those cases with a clear message, and without leaving files half-changed, is part of writing a polite script. A script that crashes messily on unexpected input, or that leaves a mess behind when interrupted, will quickly lose the trust of its users.

For scripts that will be used by others, documenting whether the script is interactive or unattended, and which mode is the default, prevents the frustration of a script that unexpectedly waits for input in the middle of an automated workflow.

Check: Why does an interactive prompt cause a problem for a script run by a scheduler?

Section 40

Dates and times appear in almost every automation task, from naming backup folders to deciding which files are old enough to archive, and they are a frequent source of subtle bugs. The difficulty is that dates and times are not as simple as they look. Time zones, daylight saving changes, and different formats for writing the same date all conspire to confuse scripts that treat dates as simple text. Handling dates with proper date and time tools, rather than string tricks, is the first step to getting them right.

A key distinction is between a date, which is a calendar day, and a datetime, which is a moment in time. A script that archives files older than thirty days needs datetimes, because it compares moments. A script that names a report by the day it covers needs a date. Mixing the two, or comparing a date with a datetime, leads to errors or to off-by-one mistakes, where a file is archived a day early or late. Being explicit about which you need, and converting carefully between them, prevents much confusion.

Time zones deserve special attention. A moment in time can be described in many time zones, and the same clock reading means different moments in different zones. A script that runs on computers in different zones, or that processes data from different zones, should store and compare times in a consistent way, preferably using a standard reference such as Coordinated Universal Time, and converting to local time only for display. That discipline avoids the classic bug where a report dated today in one zone is dated yesterday in another.

Formatting dates for filenames and reports is another practical matter. A format that sorts correctly as text, such as year-month-day, is invaluable for filenames, because it means that alphabetical order matches chronological order. A format that mixes day and month in an ambiguous way, such as writing the third of April as 03-04, can be misread depending on local conventions. Choosing an unambiguous, sortable format for machine-generated names, and a clear, human-friendly format for reports, serves both purposes well.

Finally, be careful with daylight saving transitions and with arithmetic on dates. Adding twenty-four hours to a moment is not always the same as adding one calendar day, when a clock change intervenes. Date tools handle these details correctly, if you use them, while manual arithmetic often does not. Testing date logic around transitions, and around month and year boundaries, catches errors that a test in the middle of a month would miss. Dates reward careful handling, and scripts that handle them well are noticeably more reliable.

When a script parses dates from text, it should also handle invalid or ambiguous dates gracefully, reporting them for review instead of guessing, because a guessed date can silently corrupt a report or cause a file to be archived at the wrong time.

Check: Why is a year-month-day format useful for filenames that include dates?

Section 41

Monitoring a script's runs is how you know, over time, that automation is still working. A script that ran well last month may have started failing this week, because its input changed or a folder moved. If nobody checks, the failure may go unnoticed until someone needs the output and finds it missing or stale. Monitoring does not need to be elaborate. It can be as simple as checking the log file on a regular schedule, or as structured as a summary that reports the last successful run and any recent failures.

One useful monitoring habit is to look for absence, not just for errors. A scheduled report that does not arrive is itself a signal, even if no error was logged. That is why many workflows include a check that expected output was produced, with the right date and a plausible size. A report file that is empty, or that is much smaller than usual, deserves investigation, even if the script reported success. Checking for absence and for implausible output catches problems that error logs alone would miss.

Counts and trends are also informative. If a script usually processes a hundred files a day, and suddenly processes ten, something has changed, even if the script itself did not fail. That change might be legitimate, such as a holiday reducing activity, or it might indicate a problem, such as a folder that is no longer being filled. A log that records counts makes such trends visible, and a person who reviews those counts occasionally can spot a shift before it becomes a crisis. Automation should reduce routine work, not remove human oversight entirely.

Alerts can support monitoring, but they should be used sparingly, as discussed earlier. An alert for a failed run is appropriate. An alert for every successful run quickly becomes noise, and noise is ignored. A good monitoring setup therefore distinguishes between information that can be checked on a schedule, and problems that need immediate attention. Logs serve the first purpose, alerts serve the second, and both are needed for a healthy automated workflow.

Finally, monitoring includes reviewing the automation itself from time to time. Ask whether the script is still needed, whether its input and output are still used, and whether its schedule still makes sense. Workflows evolve, and a script that was once essential may become redundant, or may need updating to match a new process. Regular review keeps automation aligned with real work, rather than letting it drift into a set of forgotten jobs that nobody fully understands.

A simple weekly check, even just glancing at the most recent log entries and confirming that output files are fresh, is often enough to catch problems early before they affect anyone else.

Check: Why is a missing expected output itself a useful monitoring signal?

Section 42

Refactoring means restructuring code without changing what it does, in order to make it clearer, simpler, or easier to maintain. For scripts, refactoring often starts when you notice duplication, or when a function has grown so long that it is hard to follow. The process is deliberate: you make a small change, test that the script still produces the same output, and then make the next change. That cycle of small, tested steps is what makes refactoring safe, even for scripts that are already in use.

A common refactoring is to extract a function. When a block of code appears twice, or when a long function contains a distinct step that could stand alone, moving that step into its own function, with a clear name, improves readability. The caller becomes shorter and easier to follow, and the extracted function can be tested on its own. Extracting functions is not about following a rule for its own sake; it is about making the structure of the script visible, so that a reader can see the main steps without being distracted by details.

Another common refactoring is to rename things. A variable called d might have made sense when the script was short, but as the script grows, a name such as destination_folder becomes far clearer. Renaming is low-risk if done carefully, and modern editors can rename consistently across a file. The benefit is immediate: code with clear names needs fewer comments, and errors become easier to spot, because a reader can tell at a glance whether a value is being used for its intended purpose.

Simplifying conditionals is also valuable. A long chain of nested conditions is hard to follow and easy to get wrong. Replacing it with an early return, a lookup, or a clearly named helper function can make the logic transparent. The goal is not to make the code shorter for its own sake, but to make the decision it embodies obvious. When a reader can see the rule at a glance, they are less likely to break it when making a later change.

Finally, refactor in small steps, and test after each step. Refactoring a whole script at once, without testing, risks introducing errors that are hard to trace, because many things changed at the same time. Small steps, each verified against sample data, keep the script working throughout the process. That discipline turns refactoring from a risky rewrite into a routine part of maintaining a script, and it is a habit that serves every larger program you will write.

Refactoring can also improve performance in passing, for example by replacing repeated work with a single calculation, but its main purpose remains clarity, so speed gains should be treated as a welcome side effect rather than the goal.

Check: Why should refactoring be done in small, tested steps rather than all at once?

Section 43

Python's standard library is a collection of modules that come with Python, covering tasks from working with files and dates to handling data formats and running other programs. For automation, the standard library is often all you need. Using it, rather than reaching immediately for an external package, has several advantages. It means your script has no extra dependencies to install, it will work wherever Python is installed, and it uses tools that are well documented and widely understood. That simplicity is valuable, especially for scripts that will be shared or run on different computers.

Learning to navigate the standard library is a skill in itself. You do not need to memorise every module, but you should know the areas it covers, and how to find the right module for a task. When you face a new automation problem, a useful first question is whether the standard library already provides a tool for it. Checking the library's documentation, or searching for a standard solution, often reveals a module that handles the task reliably, including edge cases you might not have considered. That habit saves time and avoids reinventing tools that already exist.

There are times when an external package is the right choice, such as when a task requires a format or a service that the standard library does not cover well. In those cases, choosing a well-maintained package, documenting it as a requirement, and pinning its version in a requirements file keeps the script reproducible. The decision should be deliberate, weighing the benefit of the package against the cost of an extra dependency. A script that depends on many external packages is harder to set up and more fragile when those packages change, so each dependency should earn its place.

It is also worth learning the standard library's conventions. Many of its modules follow similar patterns for opening files, handling errors, and returning results, so learning one module makes the next easier. That consistency is a deliberate design choice, and it rewards the time spent exploring the library. As you build more scripts, you will develop a mental map of which modules you reach for most often, and that map will speed up every future automation task.

Finally, remember that the standard library evolves, but slowly and carefully, which is a strength for automation. A script written with standard library tools is likely to keep working for a long time, because those tools are maintained with backward compatibility in mind. That stability, combined with the absence of extra dependencies, is why experienced Python developers often start with the standard library, and add external packages only when the task genuinely requires them.

When you are unsure which module to use, a short experiment on sample data, comparing a standard library approach with an alternative, usually reveals which one is clearer and easier to maintain for your particular task.

Check: Why is using the standard library often preferable to adding an external package for a script?

Section 44

Knowing when not to automate is as important as knowing how to automate. Some tasks are poor candidates, even though they are repetitive, because automating them would cost more than it saves, or because the human element is the point. A task that changes every time, that requires judgement about ambiguous cases, or that is done so rarely that the script would be forgotten before it is used again, is often better done by hand. Recognising those cases saves effort and avoids building scripts that nobody trusts or maintains.

One warning sign is a task whose steps cannot be stated clearly. If you cannot write down the steps in a way that another person could follow without asking questions, a script will struggle too, because it needs precise rules. In such cases, it may help to do the task by hand a few more times, paying attention to the decisions you make, until the pattern becomes clear. Sometimes that practice reveals a rule that can be automated. Sometimes it reveals that the task genuinely needs human judgement each time, and that automating it would produce poor results.

Another warning sign is a task where mistakes are costly and hard to detect. Automating such a task magnifies the impact of any error in the script, because the script may apply the error to many items before anyone notices. If the task involves irreversible actions, such as deleting data or sending messages, and if the benefit of automation is small, the risk may outweigh the reward. In those cases, a script that assists a person, by preparing data or suggesting actions, while leaving the final decision to the person, is often a safer design than full automation.

Cost is also a practical consideration. Writing, testing, and maintaining a script takes time, and that time must be repaid by the time the script saves, or by the reduction in errors it provides. A script that saves five minutes a month, but takes a day to write and needs regular maintenance, is a poor investment, unless it also teaches you a valuable skill or prevents a serious error. Being honest about the cost and benefit, rather than automating for its own sake, is part of professional judgement.

Finally, remember that automation changes work, not just speeds it up. A task that is automated may become invisible, and the knowledge involved in doing it by hand may be lost. If that knowledge is valuable, for example because it helps you notice problems, keeping the task partly manual may be wise. Automation should serve the work and the people doing it, not replace human understanding where that understanding matters.

If you do decide to automate a borderline task, start with a small, reversible version, test it thoroughly, and keep a human check in place until the script has proved itself over several runs.

Check: What is a warning sign that a task's steps are not yet clear enough to automate?

Section 45

Over time, a person who writes many scripts builds a personal toolkit: a collection of small, trusted tools that solve recurring problems. That toolkit might include a script that organises downloads, one that summarises CSV files, one that backs up a folder, and a few helper functions that appear in many scripts. Building such a toolkit deliberately, rather than accumulating scripts haphazardly, makes everyday work smoother, because you can reach for a known tool instead of starting from scratch each time.

A good toolkit starts with the scripts you actually use. When you find yourself solving the same problem for the third time, that is a signal to turn the solution into a reusable tool, with clear documentation and a sensible name, and to keep it in a known location. Tools that are hard to find will not be used, no matter how good they are, so organising them in a single folder, with a short index or readme describing each one, is part of building the toolkit. That organisation also helps you notice duplication, where two tools do almost the same job and could be combined.

Each tool in a toolkit should be small and focused, following the principles of this chapter. It should do one job well, report what it did, handle errors gracefully, and be safe to rerun. A tool that meets those standards can be trusted, and trust is what makes a toolkit useful. A collection of half-finished scripts, each with quirks and undocumented behaviour, is not a toolkit; it is a pile of code that will cause confusion when you need it most.

Maintenance applies to a toolkit as a whole. Periodically, review your tools, remove ones you no longer use, update ones whose inputs have changed, and improve ones that you find yourself working around. A toolkit that is pruned and updated stays useful, while one that is left to grow unchecked becomes cluttered and unreliable. That review need not be frequent; a brief check every few months is often enough, especially if you note problems as you encounter them in daily use.

Finally, a personal toolkit is also a learning record. It shows which problems you have solved, which techniques you have mastered, and which habits you have developed. Sharing tools with colleagues, where appropriate, can help others and can lead to improvements, as they suggest changes or report problems you had not seen. A toolkit built with care is therefore not just a convenience; it is evidence of growing skill in practical programming.

When you share a tool from your toolkit, include its documentation and a sample run, because a tool that works for you but cannot be understood by someone else will not be adopted or improved.

Check: Why should tools in a personal toolkit be kept in a known, organised location?

Section 46

A script template is a starting point: a file containing the structure that most of your scripts share, so that you do not have to write the same boilerplate each time. A typical template might include a header describing the script, a main function, basic argument handling, logging setup, and error handling that exits with a clear message. Starting from a template ensures that even a quick script has the basic safety and reporting features discussed in this chapter, rather than gaining them only if you remember to add them.

Templates save time, but their greater value is consistency. When all your scripts share a similar structure, reading a new script becomes easier, because you know where to look for its settings, its main logic, and its error handling. That consistency also makes maintenance easier, because a change to a common pattern can be applied across scripts in a similar way. For a team, a shared template is even more valuable, because it creates a common style that everyone recognises, reducing the time spent puzzling over unfamiliar structure.

A template should be simple. If it contains too much, it becomes a burden, and writers will either ignore it or spend more time deleting unwanted parts than they save. A good template contains only the parts that almost every script needs: a description, a way to run safely, a way to report progress, and a way to fail clearly. Optional features, such as configuration file handling or scheduling support, can be added when a particular script needs them, rather than being forced into every script from the start.

Templates should evolve. As you learn new habits, such as a better logging setup or a safer way to handle paths, update the template, so that future scripts benefit automatically. That evolution should be deliberate, however. Changing a template does not change existing scripts, which is good, because it avoids breaking working code, but it also means that older scripts may gradually diverge from the current template. That divergence is acceptable, as long as each script remains understandable on its own.

Finally, a template is not a substitute for thinking. Starting from a template does not remove the need to plan the script's logic, test it on sample data, or consider its risks. It simply removes the repetitive setup, leaving more attention for the parts that are unique to the task. Used in that spirit, a template is a practical tool that supports good habits, rather than a shortcut that replaces them.

If several people share a template, agree on its contents together, because a template imposed without discussion is less likely to be used than one that reflects the team's shared habits and needs.

Check: What is the greater value of a script template, beyond saving time?

Section 47

Debugging an automation script means finding why it does not behave as expected, and fixing it so that it does. Debugging starts with reproducing the problem reliably. If a script fails on real data but works on a sample, the difference between the two datasets is a clue. If it fails intermittently, the conditions under which it fails are a clue. Reproducing the problem on a small, controlled example, where you can see exactly what happens, is often the fastest route to understanding it, even if creating that example takes a little effort.

Once you can reproduce the problem, narrow it down. A script that processes many files might fail on one particular file; identifying that file, and then the particular line or value within it that causes the problem, turns a vague failure into a specific one. Techniques such as printing intermediate values, processing files one at a time, or temporarily simplifying the script can all help. The goal is to isolate the smallest case that still shows the problem, because a small case is easier to reason about than a large one.

Reading error messages carefully is a core debugging skill. Python's error messages, called tracebacks, show the chain of calls that led to the error, and the line where it occurred. That information often points directly at the problem, such as a file that does not exist, a value that cannot be converted, or a variable that was never set. Learning to read a traceback from the bottom up, starting with the error type and message, and then following the chain to your own code, saves a great deal of guessing.

It is also important to question your assumptions. A common debugging trap is to assume that a particular part of the script is correct, and to search everywhere else, when the fault is actually in the assumed part. Checking assumptions explicitly, by printing values or testing a function on its own, often reveals a surprise. A variable that you thought contained a list might contain a string, or a path that you thought was absolute might be relative. Those surprises are valuable, because they correct your mental model of the script.

Finally, when you fix a bug, add a test case that would have caught it. That test case, whether it is a sample file in a test folder or a note in the script's documentation, prevents the same bug from returning unnoticed after a later change. Debugging is not just about fixing the present problem; it is about making the script more robust against similar problems in the future. That forward-looking step turns a frustrating interruption into lasting improvement.

Keeping notes about bugs you have fixed, including what caused them and how you found them, also builds a personal debugging guide that makes future problems quicker to recognise and resolve.

Check: Why is reproducing a bug on a small example often the fastest route to fixing it?

Section 48

Failures are a normal part of automation, especially when scripts depend on files, networks, or other programs that may be temporarily unavailable. Handling failures gracefully means that a script responds to a problem in a way that is safe, informative, and recoverable, rather than crashing messily or producing misleading output. Graceful handling starts with anticipating the kinds of failures that are likely, such as a missing file, a full disk, or a program that returns an error, and deciding in advance how the script should respond to each.

One useful technique for temporary failures is retrying, with a pause between attempts. A network request that fails once might succeed a minute later, and a file that is temporarily locked might become available shortly. Retrying a few times, with a growing pause, can turn a transient problem into a non-event. Retrying should be bounded, however. A script that retries forever will hang, and a script that retries a destructive action, such as deleting a file, without checking whether it already succeeded, can cause harm. Retries are for safe, repeatable actions, and they should be logged, so that a pattern of retries is visible.

Another aspect of graceful failure is partial progress. A script that processes many items should save its progress as it goes, or be able to resume from where it stopped, rather than requiring a complete rerun after a failure near the end. Writing output incrementally, or recording which items have been completed, makes recovery much easier. That design also supports idempotence, discussed earlier, because a rerun can skip completed work and focus on what remains. Thinking about recovery while designing a script, rather than after it fails, leads to more resilient automation.

Cleanup on failure matters too. A script that creates temporary files, or that changes files in stages, should clean up or leave things in a consistent state if it fails partway through. Writing output to a temporary file and moving it into place only when complete is a good pattern, because a failure leaves the old output intact, rather than a half-written new file. Similarly, a script that creates a temporary folder should remove it on failure, or at least log its location, so that it does not accumulate clutter.

Finally, a graceful failure communicates clearly. It logs what went wrong, which item or step was involved, and what the user should do next, whether that is to fix an input file, free disk space, or simply rerun the script. A failure that is silent, vague, or misleading is worse than a failure that is loud and clear, because it wastes the time of the person who must fix it. Clear communication turns a failure from a mystery into a task with a known next step.

Check: Why should retries be bounded and logged, rather than unlimited and silent?

Section 49

A daily report workflow brings together many ideas from this chapter, and walking through one shows how they fit in practice. Imagine a small shop that records each day's sales in a CSV file, and wants a weekly summary showing totals by product, a list of any unusual entries, and a count of days covered. The workflow starts with a script that finds the week's CSV files, using a clear pattern, checks that each file has the expected headers, and reads them one by one, rather than loading everything at once.

As the script reads each file, it cleans the data conservatively. It trims stray spaces, converts text to numbers where needed, and flags rows that cannot be converted, writing those rows to a separate file for review instead of silently skipping them. It accumulates totals in a compact structure, counting rows and tracking which dates were covered, so that the final report can state how many days it includes. That counting matters, because a report covering six days instead of seven, without saying so, would mislead its reader.

When all files have been processed, the script generates the report. It writes a plain text summary, with a title, the week covered, totals by product, and a note about any flagged rows. It also writes a CSV version of the totals, for anyone who wants to chart them. Both files are written to a new folder named by the week, in a dated output location, and the script logs how many files it processed, how many rows it read, and how many it flagged. That log, together with the dated output, makes the report auditable: a reader can see when it was made and what it was based on.

The script is then scheduled to run at a fixed time, using absolute paths and a non-interactive mode, so that it needs no one present. Its output and errors are directed to a log file, which is checked occasionally, and its exit code signals success or failure to the scheduler. Before its first real run, it was tested on sample data, including a file with a missing value and a file with an unexpected header, and its dry-run mode was used to review its planned output locations. Those tests caught two mistakes that would otherwise have affected real reports.

This example is deliberately simple, but it illustrates the chapter's central message. Automation is not about writing the cleverest code; it is about building a workflow that is clear, testable, safe to rerun, and honest about its own limitations. A script built with those qualities becomes a reliable part of daily work, rather than a source of anxiety, and it can be maintained and improved as the shop's needs change.

Check: In the daily report example, why are flagged rows written to a separate file instead of skipped silently?

Section 50

This chapter has covered automation as a practical discipline, not just as a set of techniques. You have seen how to choose tasks worth automating, how to work safely with files and paths, how to process text and CSV data, how to schedule and log scripts, and how to handle errors, backups, and dry runs. You have also seen how to organise scripts, document them, keep them under version control, and judge when not to automate. Those habits, taken together, are what make automation trustworthy, and trust is the real product of this chapter.

Looking back, a few themes stand out. The first is safety before speed. A script that is fast but destructive is worse than a script that is slow but careful, because speed magnifies mistakes while safety contains them. Dry runs, backups, idempotence, and keeping originals untouched are all expressions of that theme, and they should become reflexes, not afterthoughts. The second theme is honesty in reporting. A script that logs what it did, counts what it skipped, and exits with a truthful code gives its users the information they need to trust it, and to fix it when it fails.

The third theme is simplicity. Small, focused scripts, with clear names and short functions, are easier to understand, test, and maintain than large, clever ones. Using the standard library, separating pure logic from file handling, and starting from a simple template all support simplicity. When a script grows beyond what scripting handles well, that is not a failure; it is a signal to rebuild it as a more structured program, carrying the same habits forward. Simplicity is a choice that must be renewed with each new feature, because complexity creeps in quietly.

As you move on, the best way to consolidate this chapter is to build a real script for a real task in your own work, however small. Choose a repetitive task you understand, write a script for it, test it on sample data, add a dry run and a log, and schedule it if appropriate. That single, working script will teach you more than reading about ten scripts, because it will confront you with the details that only real data reveals. Keep it, document it, and improve it over time, and it will become the first tool in your personal toolkit.

The next chapters build on this foundation. Command-line applications will formalise how scripts take options and arguments, packaging will show how to share your tools with others, and later chapters will address concurrency, security, performance, and clean code in greater depth. Automation is the context in which many of those topics matter, because it is where code meets real files, real schedules, and real consequences. Carry the habits from this chapter into those topics, and they will serve you well.

Check: What are the three themes that summarise this chapter's approach to automation?

Practical Task

Build a safe file-organising script in a practice folder. Create a test folder containing at least twenty sample files with varied names, including spaces, mixed capitalisation, and a few files that would collide if renamed carelessly. Write a Python script that lists the files matching a chosen pattern, prints a dry-run plan showing each old name and its proposed new name, checks for collisions, and only renames files when a separate confirmation mode is enabled. Add logging that records how many files were examined, renamed, and skipped, and test the script twice on the same test data to confirm that the second run is safe. Finally, write a short note describing one problem your dry run revealed and how you fixed it.

Chapter Questions

1. What makes a task a good candidate for automation, and what warning signs suggest it should stay manual?
2. Explain how dry runs, backups, and idempotence work together to make a file-changing script safe.
3. Why should a scheduled script use absolute paths, avoid interactive prompts, and write a clear log?
4. Describe how you would test a CSV summarising script, including the kinds of sample data you would use and what you would check in its output.
5. A script you wrote works by hand but fails when scheduled. List the likely causes you would investigate, in order, and explain each one.