Chapter 30: Data Analysis Basics

Section 1: What Data Analysis Is

Data analysis is the practice of examining data to answer a question and to support a decision with evidence rather than guesswork. A shop owner might ask which products sell best. A teacher might ask which topics students find hardest. A website owner might ask which days bring the most visitors. In each case, the data already exists in records such as sales receipts, test scores, or visit logs. Analysis turns those scattered records into a clear answer that a person can act on with confidence.

The work usually begins with a question. A good question is specific enough to guide everything that follows. Instead of asking how business is going, an analyst asks whether sales of one product category grew last month compared with the month before. That sharper question tells you which records to gather, which fields matter, and what a convincing answer would look like. Without it, you can spend hours exploring data and still have nothing useful to report.

Next comes gathering and cleaning the data. Gathering means collecting the relevant records from files, databases, or other sources. Cleaning means fixing problems that would distort the answer, such as inconsistent spellings, missing values, or numbers stored as text. This stage is often the longest part of an analysis, and it is also the most important for trust. A beautiful chart built on dirty data is still wrong, and nobody can tell from the chart alone that the underlying data was flawed.

Then comes summarising. Most datasets contain far too many individual records for anyone to read one by one. Summaries such as totals, averages, counts, and trends over time reveal the overall shape of the data. A total tells you the scale of activity. An average tells you what is typical. A trend tells you whether things are rising, falling, or staying flat. Together, these summaries let you see patterns that are invisible in raw rows.

Finally, analysis involves communicating findings clearly. A good report states the original question, describes the data and its limitations, presents the findings in plain language with honest charts, and says what the data does not show. It separates facts from interpretation, so readers can judge for themselves. It also says what should be investigated next, rather than pretending one analysis answers everything.

Python developers meet data analysis in many kinds of programs. A program might summarise user activity, check the quality of incoming data, or prepare a weekly report automatically. The concepts in this chapter apply whether you write plain Python or use a library designed for tables and charts. Those libraries are treated conceptually here, because their details change over time, while the ideas of asking, cleaning, summarising, and reporting honestly remain stable. Learn the ideas first, and any tool you pick up later will make much more sense.

Check: What is data analysis, and what are its main stages from question to communication?

Section 2: Questions Before Data

Before you open a dataset, write down the question you want it to answer. That simple habit prevents one of the most common failures in analysis: aimless exploration that produces interesting patterns but no useful answer. When you wander through data without a question, you will always find something that looks striking. A spike here, a difference there, a pattern that seems to mean something. Without a question to anchor you, you cannot tell which of those patterns matter and which are just noise.

A specific question also guides your choices about data. Consider the vague question of how business is doing. It could lead you to collect everything, summarise everything, and still decide nothing. Now compare it with a specific version: which product category grew most last month, and by how much? That version tells you which time period matters, which field defines a category, and which measure counts as growth. It saves time, because you ignore fields and records that cannot help answer it.

Writing the question down matters because questions drift. After seeing results, it is tempting to quietly change the question so that it matches a pleasing result. Perhaps the category you hoped would grow did not grow, but another one did. If you rewrite the question after the fact to celebrate the winner, you have not discovered anything. You have simply described whatever happened to look good. A written question, fixed before analysis begins, protects you from that self-deception and protects your readers from being misled by it.

You should also decide, in advance, what would count as an answer. If you ask whether a new layout increased visits, decide beforehand what size of increase would matter, and over what period you will measure it. A rise of one percent for one day is not the same as a sustained rise over a month. Deciding the standard early keeps you honest when the result is small, mixed, or disappointing. It also makes it easier to explain your conclusion to someone else.

A clear question has a second benefit: it limits what data you collect. When you know exactly which fields you need, you can avoid gathering sensitive information that your analysis does not require. Collecting less personal data is not only more efficient. It also reduces privacy risk for the people whose information appears in your dataset, and it reduces the work of protecting data you never needed in the first place.

Put this habit into practice on every dataset you touch. Write one sentence stating the question, one sentence stating what would count as an answer, and keep both beside your work. When the analysis is finished, return to those sentences and judge your findings against them. If the findings do not answer the original question, say so plainly. That honesty is what separates real analysis from a collection of attractive charts.

Check: Why should you write a specific analysis question and decide what counts as an answer before examining data?

Section 3: Datasets in Concept

A dataset is a collection of related records stored together so they can be analysed. Think of a table of daily sales, a list of survey responses, or a log of website visits. Each record describes one thing, such as one sale, one response, or one visit. The records share the same fields, so they can be compared, counted, and summarised together. Understanding a dataset before you analyse it is the first real step of any analysis, and skipping it is a reliable way to produce confident but wrong conclusions.

Datasets come from many sources. Some arrive as files, such as comma-separated values or structured JSON documents, which you may have met in earlier chapters on file handling. Others live in databases, where you retrieve selected rows with a query. Still others arrive through an interface that delivers records from another system. The source matters, because different sources have different quirks. A file exported from a spreadsheet may contain extra header rows. A database query may silently exclude records that fail a condition. An external feed may change its format without warning.

Before analysis begins, learn the basic facts about your dataset. What is its source, and when was it collected? How many records does it contain, and what fields does each record carry? What does each field mean, in plain language? A field named amount might mean money charged, money received, or money refunded, and those are very different things. If the meaning is unclear, ask the person who provided the data, or check any documentation that came with it. Guessing at meanings is a quiet source of error.

Also check completeness. Are records missing for some days, regions, or categories? Are some fields blank far more often than others? Missing records can bias results, because the data you have may not represent the whole picture. For example, sales data that covers only weekdays cannot support conclusions about weekends. A dataset that covers only one city cannot speak for a whole country. Note the time period and the coverage explicitly, and keep your conclusions inside those boundaries.

Pay special attention to who is missing from a dataset. If a survey reached only people with internet access, its results describe that group, not everyone. If a log records only successful transactions, it says nothing about failed ones. Missing groups can distort conclusions about a whole population, especially when the missing group differs systematically from the group that appears. Naming the missing groups in your report is part of honest analysis.

Finally, record the source and the date when you first examine a dataset. That note lets you, or someone else, repeat the analysis later and check whether a new version of the data changes the answer. A dataset is not just numbers. It is a snapshot taken at a particular time, from a particular place, with particular gaps, and good analysis never forgets that.

Check: What should you understand and check about a dataset before analysing it?

Section 4: Rows and Columns

Most data analysis begins with a table, and a table has a simple structure: rows and columns. Each row represents one record. In a sales table, one row is one sale. In a class list, one row is one student. In a visit log, one row is one visit. Each column represents one field of that record, such as the date, the product, the quantity, or the price. The intersection of a row and a column holds a single value, such as the quantity sold in that particular sale.

This structure is powerful because it supports a small set of operations that cover most analysis. You can filter rows to keep only those that meet a condition, such as sales in a single month. You can sort rows to see the largest or smallest values first. You can group rows by a column, such as product category, and summarise each group separately. You can compute totals, averages, and counts across rows. Once you understand rows and columns, you can predict how most analysis tools will behave, because they are all built around this same table idea.

Clear column names make this structure usable. A column named qty is ambiguous. Does it mean quantity ordered, quantity shipped, or quantity returned? A column named quantity_sold leaves far less room for doubt. Good names are descriptive, consistent, and free of hidden surprises such as mixed capitalisation or trailing spaces. When you inherit a dataset with unclear names, rename them in your working copy and document what each new name means. Future readers, including your future self, will thank you for that small effort.

Tables also hide traps that you should check for early. A file exported from a spreadsheet may contain a repeated header row in the middle of the data, or merged cells that leave blank values where a value should repeat. An analysis tool that expects one header row and one record per row will misread such a file, treating headers as data or dropping records silently. Inspect the first rows and the last rows of a loaded table, and look for anything that does not look like a normal record.

Another useful check is to compare row counts before and after cleaning. Cleaning should remove errors, not genuine records. If you start with one thousand rows and end with six hundred, and you cannot explain the drop, stop and investigate. Perhaps a filter was too aggressive, or a rule for removing duplicates removed real events that happened to look similar. A large unexplained drop is a warning sign, not a routine result.

Make a habit of noting the row count in an analysis log before cleaning begins, and again after each major cleaning step. That log turns cleaning from a mysterious transformation into a traceable process. Anyone reviewing your work can see what changed, when, and why, and you can return to an earlier stage if a later result looks suspicious. Rows and columns are simple ideas, but treating them carefully is the foundation of reliable analysis.

Check: What do rows and columns represent in tabular data, and why do clear column names matter?

Section 5: Types of Data

Every column in a dataset holds one kind of data, and the kind determines what you can meaningfully do with it. The main kinds are numbers, text, dates, and categories. Numbers can be counted, added, averaged, and compared mathematically. Prices, quantities, ages, and scores are numeric. Text fields hold words that describe a record, such as a product name or a comment. Date fields hold points in time, such as a sale date or a visit date, and they allow analysis over time. Categorical fields hold a limited set of values, such as a product category, a region, or a payment method.

These kinds matter because not every summary suits every kind. Averaging a price makes sense. Averaging a product name does not. Counting how often each category appears makes sense for categories. Finding the earliest and latest dates makes sense for dates. When you apply the wrong summary to a kind, you get a number that looks precise but means nothing. A common beginner error is to average codes that happen to be numbers, such as codes for regions, and then report the result as if it described a real quantity. It does not, because the codes are labels, not measurements.

Identifying the kind of each column is therefore a first step before analysis, not an afterthought. Go through the columns one by one and ask what kind each one is, and what summaries would be sensible for it. Write that down. This survey also reveals columns whose kind is unclear, such as a field that mixes numbers and words, or a field that stores dates in several different formats. Those columns will need cleaning before they can be analysed, and it is better to discover that early than halfway through a report.

You must also check kinds against actual values, because stored form can lie. A column that looks numeric may be stored as text, with values such as "42" in quotation marks. A tool that treats those values as text cannot average them correctly until they are converted. A date column may contain some values written one way and others written another way, so sorting it as text gives the wrong order. Checking a sample of real values in each column catches these mismatches before they corrupt your results.

Consider a small invented example. A table of practice shop sales might have a date column, a product category column, a quantity column, and a price column. The date supports trends over time. The category supports grouping and counting. The quantity and price support totals and averages. Each kind plays a different role, and the analysis is stronger because each role is respected.

Finally, remember that kinds can change the questions you can answer. If a dataset has no date field, you cannot study trends over time, however interesting that question might be. If it has no category field, you cannot compare groups. Part of understanding a dataset is accepting what it cannot tell you, and shaping your question to fit the data you actually have.

Check: What are the main types of data, and why must you identify and check them before analysis?

Section 6: Loading Data in Concept

Loading data means reading a dataset from its source into a form that analysis tools can work with. The source might be a file on disk, a table in a database, or records delivered by another system. In each case, loading is a translation step. The original storage format is converted into an in-memory table with rows and columns that you can filter, group, and summarise. That translation can go wrong in quiet ways, which is why loading is never a step to rush past without checking.

Loading from a comma-separated file involves reading rows and columns into a table. The first row usually holds column names, and each following row holds one record. Loading from a structured document format involves reading nested records, which may need flattening before they fit a flat table. Loading from a database involves running a query that selects certain rows and columns, and the result becomes your working table. Different sources, same destination: a table you can inspect and verify.

Verification after loading is essential. Check the row count first. Does it match what you expected from the source? A much smaller count may mean rows were skipped because of a formatting problem. A much larger count may mean header rows were counted as data. Next, check the column names. Are they exactly what you expected, spelled correctly, with no extra columns and none missing? A missing column means a question you planned to answer may now be unanswerable. An unexpected column may mean the source changed since the data was documented.

You should also inspect a sample of values. Look at the first few rows and the last few rows. Do numbers look like numbers, and dates look like dates? Is any text garbled, with strange characters replacing letters? Garbled text often signals an encoding mismatch, where text stored in one encoding was read in another. Encoding problems corrupt names and descriptions, and if they spread into summaries and charts, they undermine the whole report. Catching them at load time is far cheaper than catching them later.

Another check is to confirm that numeric columns were read as numbers, not as text. Text numbers cannot be summed or averaged correctly until converted. A quick test with a small calculation, or a check of the column kind in your tool, reveals this problem immediately. If conversion is needed, do it deliberately and document it as a cleaning step.

Finally, preserve the original source unchanged. Work on a copy, never on the only copy of the data. Preserving the original lets you, or anyone else, repeat the analysis from the same starting point and confirm the same result. It also protects you when a cleaning decision turns out to be wrong, because you can return to the untouched source and start again. Reproducibility begins at loading, with a clean separation between the source you received and the table you analyse.

Check: Why must you verify row counts and column names after loading, and preserve the original source?

Section 7: Cleaning Data

Cleaning data means fixing or removing problems in a dataset so that analysis results can be trusted. Real datasets are rarely tidy. They contain inconsistent spellings of the same category, such as a region written three different ways. They contain extra spaces or mixed capitalisation in text fields, so the same product appears as several different values. They contain numbers stored as text, or dates written in inconsistent formats, so tools cannot compare or sort them correctly. Cleaning addresses these problems before analysis, not after, because summaries built on messy data inherit the mess.

Documentation is the companion of cleaning. Every change should be recorded: what was changed, why, and how many records were affected. If you standardised three spellings of a category into one, write that down. If you converted a date format, write that down. If you removed rows, write down the rule you used and the count removed. This record lets another person follow your work, challenge a decision, or repeat the analysis. It also protects you, because a cleaning choice that looks obvious today may look questionable in a month, and a written reason is far better than a vague memory.

Cleaning must also be careful. Aggressive cleaning can remove genuine variation that analysis should capture. Suppose a dataset contains unusually large orders. Those orders might be errors, but they might also be genuine bulk purchases by a real customer. Deleting them automatically would hide a real pattern. The careful approach is to investigate first, fix clear errors, and preserve variation that could be genuine. When in doubt, flag the value and report it, rather than silently deleting it.

Units are a classic cleaning problem. Values recorded in different units cannot be compared or averaged meaningfully until they share one unit. A weight recorded sometimes in kilograms and sometimes in grams will produce a nonsense average if the two are mixed. The same applies to currencies, distances, and time durations. Standardising units is a cleaning step, and like every other cleaning step, it should be documented, including which unit you chose and how conversions were made.

A sensible cleaning workflow has a clear order. First, inspect the data and list the problems you find. Second, decide on a rule for each problem, and apply it consistently to the whole dataset, not case by case on a whim. Third, verify the result by rechecking counts, kinds, and samples after cleaning. Fourth, record everything in a cleaning log. That log becomes part of the analysis report, because readers deserve to know how the data they are seeing was shaped.

Remember that cleaning does not make data perfect. It makes data good enough, in known ways, for a stated question. Some problems may remain, and those should be reported as limitations rather than hidden. Honest cleaning is modest about what it fixed, explicit about what it changed, and open about what it could not fix.

Check: What kinds of problems does data cleaning fix, and why must cleaning be documented and careful?

Section 8: Missing Values

Missing values are fields that have no recorded value for a particular record. A customer record might lack an age. A sale might lack a region. A survey response might leave a question blank. Missing values are normal in real data, and they are not a minor detail. How you handle them can change your conclusions, so they deserve attention before any summary is computed.

Values go missing for different reasons, and the reason matters. Sometimes a field simply does not apply to a record. A question about a product feature does not apply to someone who never used that product. Sometimes data was never collected, because a form did not ask the question, or a sensor was offline. Sometimes data was lost or corrupted during collection or storage. These reasons lead to different handling, because data that is missing for a systematic reason may bias results in a way that random gaps do not.

The first step is always to count missing values, column by column. How many records lack a value in each field, and what proportion is that? A field missing in two percent of records is a different problem from a field missing in forty percent. Counting also reveals patterns. If missing values cluster in one region or one time period, that pattern itself is information, and it may warn you that conclusions about that region or period are weaker than conclusions about the rest of the data.

Only after counting and understanding should you choose a handling method. One option is to exclude records with missing values from a particular calculation. That is simple and honest, provided you report how many records were excluded and check that the excluded records do not differ systematically from the rest. Another option is to fill missing values with a reasonable placeholder, such as the average for a numeric field. Filling can preserve record counts, but it also invents data, and invented data can bias results if used carelessly.

Filling must therefore be documented and justified every time it is used. State which method you used, why you chose it, and how many values were filled. A reader should be able to tell which numbers came from real records and which came from your filling rule. If you cannot justify a filling method for a particular field, excluding the incomplete records from that calculation is usually the safer choice.

Consider a small invented example. Suppose a practice table of twenty shop visits lacks a purchase amount for three visits. Excluding those three from an average is transparent, provided you say the average covers seventeen visits. Filling the three gaps with the overall average would keep the count at twenty but would pull the result toward the average by construction. Neither choice is automatically wrong, but the choice must be visible in your report.

Missing values are a reminder that data is incomplete by nature. Counting them, understanding why they are missing, choosing a handling method for a stated reason, and documenting that choice are the marks of an analyst who respects both the data and the reader.

Check: Why must you count and understand missing values before choosing how to handle them?

Section 9: Duplicate Records

Duplicate records are records that appear more than once for the same real-world item or event. A sale entered twice, a survey response submitted twice, or a record imported from two sources without matching will all create duplicates. Duplicates are dangerous because they are quiet. They do not produce an error message. They simply inflate counts and totals, making activity look larger than it really was, and making averages shift toward the duplicated values.

Duplicates arise in several ways. Manual entry is a common cause, when the same form is typed twice. Combining data from multiple sources is another, when the same record exists in both sources and the merge keeps both copies. System retries can also create duplicates, when a submission is sent twice because the first attempt appeared to fail. Understanding the likely cause helps you choose the right detection method, because different causes leave different traces in the data.

Detection should happen before analysis begins. The key question is which fields define a duplicate. In a sales table, a duplicate might mean the same date, product, quantity, and amount appearing twice. In a registration list, it might mean the same person appearing under a slightly different spelling of a name. There is no universal rule, because the definition depends on what a record represents. You must decide the definition, state it, and apply it consistently.

Removal must be careful, because records that look identical may actually represent separate events. Two identical purchases on the same day might be two genuine purchases by the same customer, not a double entry. Two identical log entries might be two real visits that happened to match. Deleting genuine events distorts the data just as surely as keeping false duplicates does. When the distinction is unclear, investigate a sample of suspected duplicates against source records before deciding, and keep a record of what you removed and why.

Duplicates across combined sources deserve special care. The same record may appear with slightly different spellings or formats in different sources, so exact matching will miss it. Near-duplicates require judgement about which fields must match exactly and which may vary without indicating a separate record. A date that differs by one day might be a typing error, or it might be a different event. Document the judgement you made, so a reviewer can see the reasoning and challenge it if needed.

A practical habit is to count suspected duplicates, examine examples, decide on a rule, apply it, and then verify the result. Compare totals before and after deduplication, and confirm the change matches the number of records you intended to remove. If the change is much larger, your rule was probably too broad. Keep a short log of duplicates removed, including the rule and the count. That log turns deduplication from a hidden manipulation into a transparent, defensible step in your analysis.

Check: How do duplicates bias analysis, and why must deduplication be careful and documented?

Section 10: Outliers in Concept

An outlier is a value that lies far from most other values in a dataset. In sales data, it might be a single order many times larger than a typical order. In a list of ages, it might be a value that is impossibly high. Outliers attract attention because they stand out, and they deserve attention, but they do not deserve automatic deletion. The first task is always to understand why the value is extreme, because the reason determines what you should do with it.

Some outliers are genuine extreme values. A shop really does receive an occasional very large order. A website really does receive a surge of visits on a special day. These values are part of the story, and removing them would hide a real pattern or a real event. Other outliers are errors. A misplaced decimal point can turn a price of fifty into a price of five thousand. A unit mix-up can turn a weight in grams into a weight in kilograms. Errors like these should be corrected when the correct value can be established, not merely deleted.

Investigation is therefore the required first step. Check the value against source records where possible. A value that looks extreme in a summary may be explained by a genuine event recorded correctly in the source, such as a bulk order or a promotion day. Source checking distinguishes genuine extremes from transcription errors that should be corrected rather than removed. If no source record is available, look for supporting evidence in related fields. An extreme order amount paired with an extreme quantity is more plausible than an extreme amount paired with a tiny quantity.

Automatic removal is a common pitfall. Deleting every value beyond some threshold feels tidy, but it can hide genuine patterns and genuine problems that analysis should reveal. It also makes your results depend on a threshold you chose without examining the data. If you do decide to exclude an outlier from a particular summary, do so for a stated reason, report the exclusion, and consider showing the summary both with and without the value, so readers can see its effect.

Documentation completes the process. For each outlier you investigated, record the value, what you found, and what you decided: keep it, correct it, or exclude it from a specific calculation, with the reason. That record lets reviewers check your judgement and lets future analysts understand why a particular value was treated in a particular way.

Outliers also teach a broader lesson about summaries. Averages are sensitive to extreme values, while medians are more resistant. When a dataset contains a large outlier, comparing the average with the median reveals its influence immediately. That comparison is often more informative than either number alone, and it reminds you that a single summary rarely tells the whole story of a dataset.

Check: Why must outliers be investigated before any decision to keep, correct, or remove them?

Section 11: Descriptive Statistics in Concept

Descriptive statistics are summaries that describe a dataset in a few numbers, rather than listing every record. When a dataset contains thousands of rows, no reader can absorb it row by row. A count tells how many records exist. A total tells the overall scale. An average tells what is typical. A minimum and a maximum tell the range of values. Together, these few numbers give a first impression of the whole dataset, and they guide the deeper analysis that follows.

Computing descriptive statistics is a standard first step for every numeric column. Before you test a hypothesis or build a chart, compute the basics. How many values are there, and how many are missing? What is the total, the average, the smallest value, and the largest value? Those answers often reveal problems immediately. A count far lower than expected suggests records were lost during loading. A maximum that is impossibly large suggests an error or an outlier that needs investigation. Catching those problems early saves time and prevents misleading results later.

Descriptive statistics must always be reported with context, because a single number without context can mislead. An average sales figure means little without knowing how many sales it covers. An average based on ten records is far less reliable than an average based on ten thousand. Similarly, an average without its range hides whether values cluster tightly around the average or spread widely. A report that gives only an average invites readers to assume a precision and a consistency that the data may not support.

Context includes the number of records behind a statistic, the range of values, and the time period or population covered. It also includes the definition of the measure itself. An average of daily totals is not the same as an average of individual sales, and confusing the two is a common error. Stating the definition in plain language prevents that confusion. Readers should never have to guess what a number means.

Consider an invented example. A practice shop records daily sales for a month. Reporting that average daily sales were 120 units is a start, but it is incomplete. Adding that the figure covers thirty days, ranged from 40 to 260 units, and included one unusually high day during a promotion, gives readers a much truer picture. They can see that most days were closer to the lower end, and that the average was pulled upward by a single event.

Descriptive statistics are humble tools, and that is their strength. They do not prove causes or predict the future. They describe what was observed, in a compact and checkable form. Used with context, they build trust, because a reader can verify them against the data and can see exactly what is claimed and what is not. Every deeper analysis should start here, and every report should return here when a claim needs grounding.

Check: What are descriptive statistics, and why must they be reported with context?

Section 12: Mean, Median and Mode

Mean, median, and mode are three different ways to describe a typical value. They often agree when values cluster symmetrically, but they can differ greatly when a distribution is skewed or contains outliers. Knowing how each one is calculated, and when each one is most useful, prevents the common error of reporting a single average as if it were the whole story.

The mean is the familiar average. Add all the values together and divide by their count. For the invented values 4, 6, 8, and 10, the mean is 7, because the total is 28 and there are four values. The mean uses every value, which is both its strength and its weakness. It reflects the whole dataset, but a single extreme value can pull it far from the values most records actually take.

The median is the middle value when values are sorted in order. For 4, 6, 8, and 10, the median lies halfway between 6 and 8, which is 7. For an odd count, the median is simply the middle value. The median ignores how extreme the largest and smallest values are, except for their position in the order. That makes it resistant to outliers. If the largest value in the example changed from 10 to 1000, the mean would jump dramatically, while the median would barely move. That resistance is why the median is often more representative than the mean when outliers are present.

The mode is the most frequent value. It is the only one of the three that works naturally for categorical data, where averaging makes no sense. The most common product category, the most common region, or the most common payment method are all modes. For numeric data, the mode can also be useful, especially when values repeat often, such as shoe sizes or survey ratings. A dataset can have more than one mode, or no clear mode at all, if every value appears only once.

Choosing among them should be justified by the data and the question. Use the mean when values are roughly symmetric and every value should count. Use the median when the distribution is skewed or contains outliers that would distort the mean. Use the mode when the question is about the most common case, or when the field is categorical. In many reports, presenting two of them together is more honest than presenting one alone, because their difference itself reveals skew.

A small Python example makes the calculation concrete, using only built-in operations on invented data:

values = [4, 6, 8, 10, 100]
mean = sum(values) / len(values)
ordered = sorted(values)
median = ordered[len(ordered) // 2]
print(mean, median)

Here the mean is 25.6, while the median is 8. The gap between them immediately warns the reader that an extreme value is pulling the average upward. That warning is the point. A typical value is not a single truth to be chosen blindly. It is a summary whose choice should be explained, so readers can judge whether it represents the data fairly.

Check: How do mean, median and mode differ, and when is each the most useful typical value?

Section 13: Spread and Variability

Spread, or variability, describes how widely values in a dataset are dispersed around a typical value. Two datasets can share the same average yet differ greatly in how varied their values are. One class might have test scores clustered tightly around the average, while another class has the same average but scores ranging from very low to very high. Reporting only the average hides that difference, and decisions based on the average alone could be badly wrong for the more varied group.

The simplest measure of spread is the range, the difference between the maximum and the minimum values. The range is easy to understand, but it depends entirely on two extreme values, so a single outlier can make it misleading. Other measures summarise typical distance from the average, giving a sense of how far a usual record sits from the centre. Conceptually, a small typical distance means values cluster near the average, while a large one means they scatter widely. You do not need to memorise formulas at this stage. You need to understand what spread means and why it belongs beside every average.

Spread matters when comparing groups. Groups with the same average but very different spread may need different decisions. A delivery service whose times are consistent can promise a narrow delivery window. A service with the same average time but wild variation cannot make that promise honestly, because many customers would experience waits far longer than the average. A decision maker who sees only averages misses that one group is consistent while another varies wildly from record to record, which matters for planning and for risk.

Visual examination complements numeric spread. A chart showing the distribution of values reveals clustering, gaps, and long tails that a single spread number may hide. Two datasets can have similar spread measures yet look quite different when charted, for example when one has two clusters and the other has one. Combining visual examination with numeric summaries gives a fuller picture than either alone, and it helps you spot patterns that numbers alone would miss.

Consider invented daily sales for two practice shops. Both average 100 units per day. In the first shop, daily sales range from 90 to 110 units. In the second, they range from 20 to 220 units. The averages are identical, but the shops face very different realities. The first can plan staffing and stock with confidence. The second must prepare for quiet days and for surges. Any report that gave only the average would treat them as the same, which they plainly are not.

The habit to build is simple and non-negotiable. Always report spread with every average you present in an analysis summary. State the range, describe the typical distance from the average in plain language, and, where possible, show a chart of the distribution. Never present an average without its spread alongside it. That pairing is one of the smallest habits in this chapter and one of the most protective against misleading conclusions.

Check: Why must spread be reported alongside averages, and why examine it visually too?

Section 14: Distributions in Concept

A distribution describes how values in a dataset are spread across their possible range. It answers a question that a single average cannot: are values clustered in the centre, stretched toward one end, or split into separate groups? Knowing the shape of a distribution changes which summaries are appropriate, which charts are informative, and which conclusions are safe to draw.

Some distributions are roughly symmetric, with most values near the centre and fewer values toward either end. In such a case, the mean and the median will be close together, and either one gives a reasonable picture of a typical value. Other distributions are skewed, with most values at one end and a long tail stretching toward the other. Income data often skews in this way, with many modest values and a few very large ones. In a skewed distribution, the mean is pulled toward the tail, while the median stays near the bulk of the values, so the median is usually the more representative typical value.

Some distributions have multiple peaks, when distinct groups are mixed in one dataset. Test scores for two different classes combined into one list might show two peaks, one for each class. An overall average for such a mixed distribution describes neither group well. It sits in a valley between the peaks, where few actual records lie. Recognising multiple peaks warns you to split the data by group before summarising, rather than reporting a single number that represents nobody.

Distributions are usually examined through charts that show how many values fall in each part of the range. Such a chart makes skew and multiple peaks visible at a glance, in a way that a table of numbers does not. Before choosing a typical value for a numeric field, chart its distribution first. That order, chart first and summarise second, prevents the error of applying a summary that the shape does not support.

Subgroups deserve the same treatment. An overall distribution that looks simple may hide very different distributions within subgroups. Sales overall might look symmetric, while sales in one region skew heavily and sales in another cluster tightly. Grouping and distribution examination work together in thorough analysis: grouping reveals the subgroups, and charting each subgroup reveals whether the overall shape was hiding important differences.

A practical workflow follows from this. For each numeric field, chart the distribution, note whether it is symmetric, skewed, or multi-peaked, and then choose summaries that fit that shape. Report the shape in plain language alongside the numbers, so readers understand why you chose the median rather than the mean, or why you split the data by group. A distribution is not a decoration. It is the evidence that justifies your choice of summary, and a report that omits it asks readers to trust a number without showing them the shape behind it.

Check: What does a distribution show, and why does its shape affect which summaries are appropriate?

Section 15: Grouping Data

Grouping data means splitting a dataset into groups based on a categorical field and summarising each group separately. Instead of reporting one overall average, you report an average for each region, each product category, or each time period. Grouping is one of the most revealing operations in analysis, because overall summaries can hide differences that matter for decisions.

Consider a practice shop with overall average daily sales of 100 units. That single number sounds steady. Grouped by region, however, one region might average 160 units while another averages 40 units. The overall average hid a strong region and a struggling one, and a decision based only on the overall figure, such as equal staffing everywhere, would be wrong for both. Grouping by category can reveal the same kind of story, showing that some categories outperform others even when total sales look flat.

Grouping should be guided by the original analysis question. If the question concerns regions, group by region. If it concerns categories, group by category. Grouping by every available field in the hope of finding something interesting is a form of aimless exploration. It produces many comparisons, some of which will look striking by chance alone, and it makes it harder to tell which differences are real. A question-led grouping keeps the analysis focused and its conclusions defensible.

Group sizes must be reported alongside every grouped summary. Averages from tiny groups are unreliable, because a single unusual record can swing them dramatically. An average based on three records is not comparable to an average based on three thousand records, even if the two numbers look similar. Reporting the record count for each group lets readers judge reliability for themselves, without needing to ask for it. Small groups deserve a caution note whenever their averages are reported, and in some cases the honest choice is to combine small groups or to omit their averages entirely.

Grouping also guards against a subtle trap, sometimes called Simpson's paradox, where an overall trend reverses within subgroups. A treatment, a product, or a policy might look better overall but worse within every individual group, because the groups differ in size and composition. Checking subgroups before drawing conclusions prevents recommending an action based on a trend that does not hold for any individual group in the data. The overall figure is not wrong as arithmetic, but it is misleading as a guide to action.

In Python, grouping is conceptually simple: choose a field, split the rows by its values, and compute the same summary for each split. The mechanics depend on the tool you use, but the idea is identical everywhere. Group thoughtfully, report group sizes, flag small groups, and check whether the overall story survives when the data is split. Those four habits turn grouping from a routine operation into a reliable source of insight.

Check: Why does grouping reveal what overall summaries hide, and why report group sizes?

Section 16: Filtering Data

Filtering data means keeping only the rows that meet a condition and setting the rest aside for a particular calculation or view. A filter might keep only sales from last month, only visits from one region, or only orders above a certain amount. Filtering is essential, because most analysis questions concern a subset of the data, not the whole dataset at once. Without filtering, you cannot answer questions about a specific period, group, or threshold.

Filters should follow from the question, just as grouping does. If the question concerns last month, the filter is the date range for last month. If it concerns one category, the filter is that category. Stating the filter in plain language, before you apply it, keeps the analysis transparent. A reader should be able to reproduce the filtered set from your description alone, without guessing at boundaries. Inclusive or exclusive boundaries matter here. Sales from the first to the last day of a month is a different set from sales strictly between those dates, and the difference can change totals.

Filtered results must always be documented with their conditions and the resulting record count. A statement such as average sales were 120 units is incomplete if it hides that the average covers only a filtered subset. The complete statement gives the filter and the count: average sales were 120 units across 84 orders in the northern region during March. That version lets readers judge whether the subset is large enough to trust and whether it matches the question being asked. It also lets another analyst repeat the calculation and confirm the same result.

Filtering also interacts with missing values and duplicates. A filter applied before cleaning may behave differently from the same filter applied after cleaning, because cleaning changes which rows exist and which values they hold. For consistency, apply cleaning first, then filter the cleaned data, and document that order. If you change a filter during analysis, document the change and the reason, rather than quietly presenting results from a later filter as if they came from the first.

A common pitfall is filtering so narrowly that almost nothing remains, and then reporting averages from a handful of records as if they were robust. Another pitfall is the opposite: applying a filter, forgetting it is active, and then reporting filtered results as if they described the whole dataset. Both errors are prevented by the same habit, which is to state the filter and the count every time you report a filtered result.

In practice, filtering is often combined with grouping and sorting. You might filter to one year, group by category, and sort the groups by total sales to see which category led. Each operation adds clarity, but each one also narrows or reshapes the view, so each one should be visible in the report. A filtered, grouped, sorted result is powerful precisely because it answers a precise question. Its power depends on the reader knowing exactly which question it answers.

Check: Why must filtered results be documented with their conditions and resulting record count?

Section 17: Sorting Data

Sorting data means arranging rows in order by the values in a chosen column, from smallest to largest or from largest to smallest. Sorting is a simple operation, but it reveals things that unsorted data hides. The largest and smallest values become immediately visible. The order of categories by total or average becomes clear. Gaps, clusters, and unusual values stand out when rows are arranged rather than scattered.

Sorting reveals ranking, which is often the heart of a question. Which products sold most? Which days had the fewest visitors? Which regions grew fastest? A sorted list answers those questions directly, without requiring the reader to scan an entire table. Sorting also helps during investigation, because the extremes of a sorted column are the first place to look for outliers, errors, and interesting cases worth examining in detail.

The direction of a sort must always be stated. A list sorted from largest to smallest tells a different story at first glance from the same list sorted from smallest to largest, even though both contain the same rows. A report that says products are listed in order, without saying which order, forces readers to guess. State the column, the direction, and the measure being sorted, such as total sales descending, so the ranking is unambiguous.

Sorting is most useful when combined with filtering and grouping. Filtering narrows the data to the relevant subset, grouping summarises it by category, and sorting arranges those summaries so the most important groups appear first. For example, you might filter practice sales to a single month, group them by product category, compute total sales per category, and sort the categories from highest to lowest total. Each step has a purpose, and the final sorted table directly answers the question of which category led that month.

Sorting has limits that are worth remembering. A sorted list shows order, but it does not show how large the gaps between ranks are. The top two products might differ by one unit or by one thousand units, and the ranking alone does not tell you which. Pairing a sorted list with the actual values, or with a chart, restores that information. Sorting also says nothing about why the order came out as it did. A product that ranks first might lead because of a promotion, a price change, or a seasonal effect, and sorting alone cannot distinguish those causes.

Ties deserve a note as well. When several rows share the same value, their relative order may be arbitrary, depending on the tool and the original order of the data. If ties matter for a decision, such as awarding a prize to the top performer, state how ties were handled, or report the tied group together rather than inventing a winner. Sorting is a presentation choice as much as a calculation, and like every presentation choice in analysis, it should be stated plainly so readers are not misled by the order in which results appear.

Check: What does sorting reveal, and why state its direction and combine it with filtering and grouping?

Section 18: Aggregation in Concept

Aggregation means combining many individual records into a smaller number of summary values. Totals, counts, and averages are all aggregations. When you add up all sales in a month to get monthly revenue, you are aggregating. When you count how many visits came from each region, you are aggregating. Aggregation is what turns a long list of records into a handful of numbers that a decision maker can grasp and act on.

Aggregation usually follows grouping. You group rows by a categorical field, and then you aggregate within each group. Grouping by product category and summing sales gives total sales per category. Grouping by month and counting visits gives visits per month. The combination of grouping and aggregation is the engine of most summary tables and many charts. It is also where errors creep in most easily, because a mistake in the grouping or in the count propagates into every summary that follows.

Verification is therefore essential before trusting any total. Check the grouping field first. Are its values clean and consistent, or do spelling variations split one real group into several apparent groups? A category spelled two ways will appear as two groups, each with a partial total, and neither total will be correct. Check record counts next. Does the sum of the group counts equal the total number of records you expected? If it does not, records were lost, duplicated, or filtered in a way you did not intend. That check takes moments and catches a surprising number of errors.

Also check that the aggregation matches the question. Summing quantities answers a question about volume. Averaging quantities answers a question about typical size. Counting records answers a question about frequency. Using the wrong aggregation gives a correct calculation of the wrong thing, which is worse than an obvious error, because it looks like an answer. State the aggregation in plain language in your report, such as total sales per category, so readers know exactly what was computed.

A small invented example clarifies the idea. Suppose a practice table holds daily sales of three invented products over a week. Aggregating by product gives three totals, one per product. Aggregating by day gives seven totals, one per day. Both aggregations are valid, but they answer different questions. The first reveals which product sold most overall. The second reveals which day was busiest. Choosing the aggregation is choosing the question, and the report should make that choice explicit.

Finally, remember that aggregation hides detail by design. A total conceals the distribution behind it, including outliers and variation. That is why aggregation should be paired with checks on spread, group sizes, and distributions, as covered elsewhere in this chapter. An aggregated summary is a useful simplification, not a replacement for understanding the data beneath it, and a good analyst always knows what the summary is hiding.

Check: What is aggregation, and why must you verify grouping and record counts before trusting totals?

Section 19: Joining Datasets in Concept

Joining datasets means combining records from two tables that share a common field, so that information from both tables can be analysed together. Suppose one table holds sales, with a product code for each sale, and another table holds product details, with the same product code plus a category and a price. Joining them on the product code produces a combined table where each sale carries its category and price. Without a join, those questions that need fields from both tables cannot be answered at all.

The shared field, often called a key, is the heart of a join. Planning a join begins with choosing that field and checking that it really identifies the same thing in both tables. A product code in one table and a product name in another may look like a match, but names vary in spelling while codes usually do not. A date in one table and a month label in another may not align exactly. Checking the key means verifying that values overlap, that formats match, and that duplicates in the key do not create unexpected multiplication of rows.

Unmatched records are the central decision in any join. Some sales may refer to product codes that have no entry in the details table, perhaps because a product was discontinued or a code was mistyped. Some products in the details table may have no sales at all. You must decide, before joining, how to handle these cases. One option keeps only records that match in both tables. Another keeps all records from one table, filling missing fields from the other with blanks. Each option answers a different question and produces a different row count, so the choice must be stated in the report.

Row counts after a join deserve the same verification as row counts after loading or cleaning. If a join unexpectedly multiplies rows, the key probably contains duplicates in one table, so each matching row pairs with several rows from the other table. If a join unexpectedly drops rows, unmatched records were probably excluded by the join type you chose. Comparing counts before and after the join, and investigating any surprise, prevents totals from being inflated or deflated silently.

Joins also require care with field names. When both tables contain a column with the same name but different meanings, the combined table can confuse them. Renaming columns before joining, and documenting the renaming, avoids that trap. Similarly, when a join adds many columns, the combined table can become wide and hard to read, so selecting only the fields needed for the question keeps the analysis focused.

In concept, a join is a way of enriching data by bringing related information together. In practice, it is also a place where silent errors thrive, because a join that runs without an error message can still be wrong. Plan the shared field, check its values, decide how unmatched records will be handled, verify row counts, and document all of it. Those steps turn joining from a risky merge into a trustworthy part of an analysis.

Check: What is a join, and why must you plan the shared field and decide how to handle unmatched records?

Section 20: Pivot Tables in Concept

A pivot table is a summary table that rearranges grouped data so that one categorical field forms the rows, another forms the columns, and the cells hold an aggregated value, such as a total or an average. Imagine practice sales grouped by region and by product category. A pivot table might list regions down the side, categories across the top, and total sales in each cell, with row totals and column totals at the edges. That layout lets you compare regions and categories at a glance, in a way that a long grouped list does not.

Pivot tables are useful because they show two dimensions at once. A grouped list shows one dimension well, such as totals by category, but comparing that breakdown across regions requires flipping between several lists. A pivot table places the comparison in a single view. You can see which category leads overall, which region leads overall, and whether a category that leads in one region lags in another. Patterns that involve the interaction of two fields become visible.

Building a pivot table involves three choices, and each should be stated in the report. The row field defines the groups down the side. The column field defines the groups across the top. The value field, together with its aggregation, defines what each cell contains. Total sales, average sale size, and count of sales are all possible values, and they answer different questions. A pivot table of totals and a pivot table of averages can tell quite different stories from the same data, so the aggregation must never be left ambiguous.

Empty cells require careful interpretation. An empty cell means that no records existed for that combination of row and column. It does not necessarily mean the value was zero in a meaningful sense, and it does not mean the combination is impossible. Perhaps a product was not sold in a region during the period covered, or perhaps data for that region was missing. Treating every empty cell as zero can distort row and column totals and can suggest conclusions about combinations that were never observed. When empty cells appear, note them, explain what they represent in your dataset, and decide explicitly whether they should be treated as zero for a particular total or left as missing.

Pivot tables also inherit the cautions of grouping. Small groups produce unreliable cell values, and a cell based on two records should not be compared confidently with a cell based on two thousand. Reporting the underlying counts, or at least flagging small cells, keeps the comparison honest. Totals at the edges should be checked against overall totals, to confirm that no records were lost or duplicated in the rearrangement.

Used well, a pivot table is a compact, readable summary of a two-dimensional question. Used carelessly, it can hide small samples behind neat cells and turn missing data into apparent zeros. State the row field, the column field, the aggregation, and the meaning of empty cells, and the pivot table becomes a trustworthy tool rather than a polished trap.

Check: What is a pivot table, and why must empty cells be interpreted carefully?

Section 21: Charts in Concept

Charts turn numbers into pictures, and pictures are often easier to grasp than tables. A chart is not decoration for a report. It is a way of showing a pattern, a comparison, or a trend so that readers can see it directly, without performing mental arithmetic on rows of figures. A rising trend, a dominant category, or a skewed distribution can be recognised in a chart within seconds, while the same insight might take minutes to extract from a table.

Every chart exists to serve a question. A chart that shows sales over time serves a question about trends. A chart that compares categories serves a question about which group leads. A chart that shows a distribution serves a question about shape and spread. Choosing a chart therefore begins with the question, not with the available chart types. When the question is unclear, the chart will be unclear too, however polished it looks.

Clear labelling is what makes a chart trustworthy. Every chart needs a title that states what it shows, in plain language. Every axis needs a label that names the quantity and its unit, such as sales in units or revenue in a stated currency. Every series or category needs a legend or a direct label, so readers know which colour or bar represents which group. Without these labels, a chart can be read in several ways, and at least some readers will read it wrongly. Labels are not optional extras. They are part of the chart's meaning.

Charts also need context. A chart of sales over a single month cannot support claims about a whole year. A chart based on a filtered subset should say so, either in its title or in a note beneath it. A chart that omits a relevant group, or that starts its axis at a value that exaggerates small differences, misleads even when its numbers are correct. Honest labelling includes stating the period covered, the population covered, and any filtering that shaped the data shown.

A good habit is to read your own chart as a stranger would. Cover the surrounding text and ask what the chart alone communicates. Does the title match the picture? Do the axes make sense? Could a reader mistake one series for another? If any answer is unclear, revise the labels before sharing the chart. Charts are often copied out of reports and shared on their own, so a chart that only makes sense with its surrounding paragraphs is a fragile chart.

Finally, remember that a chart is a summary, and summaries hide detail. A chart of monthly totals hides daily variation. A chart of averages hides spread. Pairing a chart with a short note about the underlying data, including the record count and any important limitations, keeps the picture honest. A well-labelled chart, built for a clear question and accompanied by its context, is one of the most effective tools in analysis, because it lets readers see the evidence for themselves rather than merely trusting the analyst's words.

Check: What are charts for, and why must every chart be clearly labelled with titles, axes and units?

Section 22: Choosing the Right Chart

Choosing the right chart means matching the chart type to the data type and to the question being asked. Different chart types suit different jobs, and using the wrong type can obscure the very pattern you want to show. The choice is therefore a reasoning step, not a matter of taste or of picking the most attractive option from a menu.

Comparisons between categories are usually shown with bars, where the length of each bar represents a value for one category. Bars make it easy to see which category is largest, which is smallest, and how large the gaps are. Trends over time are usually shown with lines, where the line connects values in time order and its slope shows whether values are rising, falling, or flat. Proportions of a whole are sometimes shown with parts of a circle, but only when the number of parts is small, because many thin slices become hard to read. Distributions are shown with charts that count values in each part of the range, revealing shape, skew, and peaks.

Matching the chart to the data type is the first rule. Numeric values over time suit a line. Counts or totals by category suit bars. A single numeric field's distribution suits a counted-range chart. Trying to force a categorical field into a line chart, or a time series into a set of unrelated bars, usually produces a confusing picture. When the match is right, the chart feels obvious, and readers grasp it without explanation.

Several distortions must be avoided, whatever chart type you choose. Starting a bar chart's axis above zero exaggerates differences, because bar lengths no longer represent values proportionally. A bar twice as long should mean a value twice as large, and a truncated axis breaks that promise. Cherry-picking a short time period that shows a favourable trend, while omitting the longer history, is another distortion. So is using inconsistent scales across charts that are meant to be compared, or using three-dimensional effects that make values harder to read. Each distortion makes a chart look more dramatic or more favourable than the data warrants.

Colour and annotation also need care. Colour should distinguish categories clearly, but it should not imply an order or a judgement that the data does not support. Direct labels on bars or points are often clearer than a legend that forces readers to match colours back and forth. Annotations can highlight a genuine event, such as a promotion, but they should not be used to suggest a cause that the data has not established.

A practical test is to sketch the chart by hand, or in words, before building it. State what the reader should see in it, and check whether the chosen chart type actually shows that. If the insight you want is a ranking, a sorted bar chart will show it. If the insight is a trend, a line chart will show it. If no chart type shows the insight clearly, the problem may be the question, not the chart, and refining the question is the honest next step. A well-chosen chart does not need decoration. Its clarity is its strength.

Check: How do you match chart type to data type and question, and what distortions must you avoid?

Section 23: Trends Over Time

A trend is a sustained direction of movement over time, such as sales rising gradually across several months, or visits falling steadily across several weeks. Trends matter because decisions are usually about the future, and a sustained direction is more informative than a single day's figure. A trend is identified by looking at a series of values in time order and judging whether the overall movement points upward, downward, or remains roughly flat, despite short-term ups and downs.

Seasonal patterns are a related but distinct idea. A seasonal pattern repeats at regular intervals, such as higher sales every December, or more visits every weekend. Seasonal patterns are predictable in timing, even if their exact size varies. Recognising seasonality prevents a common error, which is to mistake a seasonal rise for a new trend, or a seasonal fall for a decline. Comparing a month with the same month a year earlier, rather than with the previous month, often gives a fairer picture when seasonality is present.

Short-term fluctuations must be interpreted carefully. Daily or weekly values bounce around for many reasons, including weather, one-off events, random variation, and measurement quirks. A single high day does not establish a trend, and a single low day does not establish a decline. Analysts often smooth short-term noise by looking at averages over a longer window, such as a weekly or monthly average, which reveals the underlying direction more clearly. Smoothing is a presentation choice, and the window used should be stated, because different windows can suggest different stories.

Time order is essential to trend analysis. Values must be placed in their true chronological sequence, with gaps handled explicitly. If data is missing for a week, the chart should show the gap, not connect the surrounding points as if nothing were missing. Connecting across a gap implies continuity that was not observed. Similarly, if the way a quantity was measured changed partway through the series, that change should be marked, because a jump caused by a measurement change is not a real trend.

Context also shapes trend interpretation. A rising trend during a promotion may reflect the promotion rather than lasting growth. A falling trend during a period when data collection was incomplete may reflect missing records rather than falling activity. Before attributing a trend to a cause, check for events, promotions, measurement changes, and gaps that coincide with the movement. Correlation over time is not proof of cause, a point this chapter returns to in detail.

Reporting a trend honestly means stating the period covered, the direction observed, and the strength of the evidence. A trend described as steady growth over twelve months, based on monthly totals with no major gaps, is a stronger claim than growth suggested by two recent weeks. State the period, show the chart, note any seasonal pattern, and acknowledge short-term fluctuation. That careful language lets readers distinguish a robust trend from a hopeful reading of noisy data, and it protects decisions from being based on a blip mistaken for a direction.

Check: What are trends and seasonal patterns, and why must short-term fluctuations be interpreted carefully?

Section 24: Correlation in Concept

Correlation describes a relationship between two variables, where values of one tend to move with values of the other. When one variable rises and the other tends to rise as well, the correlation is positive. When one rises and the other tends to fall, the correlation is negative. When no consistent pattern exists, the variables are said to have little or no correlation. Correlation is a descriptive summary of a pattern in observed data. It does not, by itself, explain why the pattern exists.

Correlations are often examined with a scatter chart, where each point represents one record, with one variable on each axis. A cloud of points sloping upward suggests a positive relationship. A cloud sloping downward suggests a negative one. A cloud with no slope suggests little relationship. The tighter the points cluster around a sloping pattern, the stronger the apparent relationship. The looser the cloud, the weaker it is. Visual inspection of such a chart is an essential first step, because a single summary number can hide patterns that the chart reveals immediately.

Outliers must be checked before trusting an apparent relationship. A single extreme point can create the appearance of a strong correlation where none exists among the bulk of the data, or it can hide a relationship that is otherwise clear. Imagine a set of points with no pattern, plus one faraway point that happens to align with a slope. A summary of correlation computed over all points would suggest a relationship, but removing that one point would reveal that the remaining points show none. Checking outliers, and reporting results both with and without them when they matter, prevents that error.

Correlation can also be distorted by grouping. A relationship that appears in combined data may disappear, or even reverse, when the data is split by group, just as overall trends can reverse within subgroups. A relationship that holds within each group separately is more convincing than one that appears only when groups are mixed. Grouped examination is therefore a useful check before reporting a correlation as a general finding.

Range matters too. A correlation computed over a narrow range of values may look weak, even when a stronger relationship exists over a wider range. Conversely, a correlation computed over a range that includes two distinct clusters may reflect the difference between clusters rather than a smooth relationship within either one. Stating the range covered, and showing the chart, lets readers judge whether the correlation describes the data as a whole or only a part of it.

Finally, correlation is a starting point for investigation, not a conclusion. It suggests that two variables are related in the observed data, and it invites further questions about why. Those questions may lead to a plausible explanation, to a third variable that influences both, or to a coincidence. The next section examines why correlation alone never proves causation, and how reporting language should reflect that limit. For now, the habit to build is to chart the relationship, check outliers and groups, state the range, and describe the pattern as an association in the data, nothing more.

Check: What is correlation, and why must outliers be checked before trusting an apparent relationship?

Section 25: Correlation Is Not Causation

Correlation does not prove causation. That sentence is short, but it guards against one of the most common and costly errors in analysis. Two variables can move together for many reasons other than one causing the other. One variable might cause the other, but it might also be the reverse, or both might be influenced by a third variable, or the association might be a coincidence in the particular data examined. Observing that two things move together does not, by itself, distinguish among those possibilities.

Third variables are a frequent explanation. Consider invented practice data in which ice cream sales and sunburn cases rise together during summer months. Neither causes the other. Both rise because warm weather increases both. An analyst who reported that ice cream causes sunburn, based only on the correlation, would be laughed at in that example, but the same error in a business or policy setting is less obvious and far more consequential. A promotion, a season, a price change, or a change in measurement can all act as a third variable that creates an apparent relationship.

Reverse causation is another possibility. A correlation between advertising spending and sales might mean advertising drives sales, but it might also mean that higher sales provide a larger budget for advertising. Data alone, without knowledge of timing and mechanism, often cannot tell which direction is correct. Timing can help, if a change in one variable consistently precedes a change in the other, but even timing is not proof, because both might follow a third event that happened earlier still.

Coincidence also deserves respect, especially in small datasets or when many pairs of variables are examined. If you test enough pairs, some will appear correlated by chance alone. That is why a correlation found by searching through many variables, without a prior question or hypothesis, should be treated as a suggestion for further investigation, not as a discovery. The more pairs you examined, the more cautious you should be.

Reporting language must reflect these limits. Say that two variables were associated in the data, or that they tended to move together during the period examined. Do not say that one caused, boosted, reduced, or led to the other, unless you have evidence beyond correlation, such as a controlled comparison or a well-understood mechanism. Honest language also states the period, the population, and any third variables you considered. It invites readers to weigh the evidence, rather than presenting a suggestion as a fact.

Decisions should follow the same caution. Acting as if a correlation were a cause can waste money and miss the real lever. If a correlation suggests a promising intervention, the prudent next step is to test it deliberately, for example by comparing a group that receives the intervention with a similar group that does not, over a defined period. Correlation is valuable precisely because it points to where such tests are worth running. Its value lies in guiding investigation, not in replacing it.

Check: Why does correlation not prove causation, and how should reporting language reflect that?

Section 26: Time Series in Concept

A time series is a sequence of values recorded in time order, with each value attached to a particular moment or period. Daily sales, weekly visits, and monthly totals are all time series. Time series analysis studies how such values change over time, looking for trends, seasonal patterns, and unusual events. Because time order carries meaning, time series data must be handled with care that other datasets do not require. Shuffling the rows of a time series destroys information, while shuffling the rows of an ordinary table does not.

Gaps are a central concern. Data collection sometimes stops for a period, whether because a system was offline, a form was unavailable, or records were lost. A gap in a time series is not the same as a value of zero. Zero means nothing happened. A gap means nothing was recorded, and activity during the gap is unknown. Treating gaps as zeros creates false drops in a chart and false conclusions about decline. Gaps should be shown as gaps, counted, and reported, so readers know which periods are unobserved.

Measurement changes are equally important. If the definition of a quantity changes partway through a series, values before and after the change are not directly comparable. A shop that begins counting online orders alongside in-store orders will show a jump in total sales that reflects the counting change, not a real surge in business. A website that changes how it counts a visit will show a similar break. When a measurement change is known, mark it on the chart and in the report, and consider analysing the periods before and after separately.

Irregular spacing also needs attention. Values recorded daily can be compared day by day, but values recorded at irregular intervals cannot be treated as if they were evenly spaced. Averages over calendar months with different numbers of days, or over weeks that straddle month boundaries, need careful definition. Stating the interval for each value, and the rule used to aggregate values into periods, prevents subtle errors that accumulate over a long series.

Seasonality, trends, and noise all appear in time series, and distinguishing them takes time and a long enough series. A series covering only a few weeks cannot reveal an annual seasonal pattern. A series covering a single season cannot show whether a rise is seasonal or lasting. The length of the series limits the conclusions it can support, and that limit should be stated plainly. A three-month series can support claims about those three months, not about a yearly cycle.

Finally, time series invite forecasting, but this chapter treats forecasting only conceptually. Predicting future values from past patterns assumes that the patterns will continue, an assumption that events, promotions, and changes in conditions can break. Any forecast should be presented with its uncertainty, its assumptions, and its limits. The first task of time series analysis is simpler and more valuable: describe what happened, in order, with gaps and measurement changes made visible, so that any claim about trends rests on a clear and honest picture of the past.

Check: What is a time series, and why must gaps and measurement changes be checked explicitly?

Section 27: Text Data Basics

Text data consists of words, and basic text analysis begins with counting and grouping those words. A set of comments, reviews, survey answers, or messages can be summarised by counting how often each word appears, which words appear together, and how long responses tend to be. Those simple counts often reveal the main topics people mention, the language they use, and whether responses are brief or detailed. Text analysis at this level does not require understanding every nuance of language. It requires careful counting and honest interpretation.

Cleaning comes first, just as it does for numeric data, and it shapes everything that follows. Text fields often contain inconsistent capitalisation, extra spaces, punctuation differences, and spelling variations. The same word written with different capitalisation will be counted as two different words unless capitalisation is standardised. A word with a trailing space or a different punctuation mark may also be counted separately. Standardising these surface differences before counting is essential, and the rules used should be documented so the counts can be reproduced.

Grouping words is the next step. Counting individual words is useful, but grouping related words into themes is often more informative. For example, comments mentioning price, cost, and expensive might all belong to a theme about affordability. Grouping can be done by hand for small datasets, by reading responses and assigning themes, or with simple rules for larger ones. Either way, the grouping rules should be stated, because different groupings can lead to different conclusions, and readers deserve to see how themes were defined.

Text analysis must also respect what text cannot show. A word count shows frequency, not importance. A topic mentioned rarely might still matter greatly, especially if it concerns a serious problem that only a few people experienced. A topic mentioned often might reflect a prompt in the question rather than a genuine concern. Frequency is a starting point for reading examples, not a replacement for reading them. Always read a sample of the actual responses behind each count or theme, to check that the summary matches what people actually said.

Privacy is a special concern with text data, because words can identify people even when names are removed. A distinctive story, a rare combination of details, or a quoted phrase can reveal who wrote a response. When reporting text findings, use aggregated counts and paraphrased themes rather than long verbatim quotations, unless consent and safety allow quotation. Remove identifying details before sharing examples, and consider whether a small dataset makes identification likely even after names are removed.

Finally, document the text workflow as you would any other analysis. State the source of the text, the period covered, the cleaning rules, the grouping rules, and the counts behind each theme. Text analysis is often presented as if it were objective counting, but its choices, from standardisation to grouping, involve judgement. Making those choices visible is what makes text findings trustworthy, and it lets readers distinguish a careful summary of what people said from a selective reading that supports a preferred story.

Check: How does basic text analysis begin, and why must text be cleaned and documented first?

Section 28: Categorical Data Analysis

Categorical data consists of values drawn from a limited set, such as product categories, regions, payment methods, or survey answers. Categorical analysis begins with counting how often each value appears. Those counts reveal which categories dominate, which are rare, and how the distribution of categories compares across groups or time periods. A table of counts, often paired with a bar chart, is the foundation of categorical analysis, and it answers questions that averages cannot address.

Counting seems trivial, but it requires the same cleaning discipline as numeric analysis. The same category spelled two ways will be counted as two categories, distorting the picture. Extra spaces, mixed capitalisation, and inconsistent abbreviations all split real categories into fragments. Standardising category values before counting, and documenting the standardisation, is therefore a required first step. A count table is only as trustworthy as the category definitions behind it.

Rare categories need explicit handling. A dataset may contain a long tail of categories that appear only once or twice each. Reporting every rare category separately produces a long, noisy table that obscures the main pattern. Grouping rare categories into a combined other category is a common solution, but it must be done transparently. State the threshold used, such as categories with fewer than ten records, and state how many records and how many distinct categories were combined. A reader should be able to tell how much of the data sits in the other group, and should not mistake a combined total for a single real category.

Missing category values deserve the same attention. A blank category is not the same as a category called other, and it should not be silently merged into other without explanation. Count missing values separately at first, investigate why they are missing, and then decide, with documentation, whether to report them as a separate group or to exclude them from a particular comparison. That decision affects percentages, because percentages computed over different denominators are not comparable.

Proportions and percentages are the usual way to present categorical results, but they must be paired with counts. A category that accounts for fifty percent of a group sounds dominant, but if the group contains only four records, that percentage rests on two records. Reporting both the percentage and the count, such as 2 of 4 records, lets readers judge reliability. This pairing echoes the rule for grouped summaries generally: never report a proportion without the numbers behind it.

Comparing categorical distributions across groups is where categorical analysis becomes most informative. Does the mix of categories differ by region, by month, or by customer group? A side-by-side comparison of counts or percentages can reveal shifts that overall totals hide. As with grouped numeric analysis, check group sizes before drawing conclusions, and be cautious about small groups. Categorical analysis is simple in its tools and rich in its pitfalls, and its reliability comes from clean categories, transparent handling of rare and missing values, and counts shown alongside every percentage.

Check: How does categorical analysis begin, and why must rare categories be handled explicitly?

Section 29: Sampling in Concept

Sampling means studying a subset of a larger population in order to learn about the whole. When a dataset is too large to examine completely, or when collecting data from everyone is impractical, a sample offers a practical alternative. A sample can support conclusions about the population it came from, but only if it was chosen well. A poorly chosen sample can mislead more than no data at all, because it carries the appearance of evidence without its substance.

Random sampling is the ideal where possible. In a random sample, every member of the population has a known and fair chance of being selected, and selection does not depend on convenience, willingness, or the analyst's judgement. Randomness protects against systematic bias, because it prevents the sample from over-representing an easy-to-reach group and under-representing a hard-to-reach one. A sample of website visitors recruited only from a help page, for example, will over-represent visitors who needed help, and conclusions from it will not describe all visitors.

Where random sampling is not possible, the sampling method must be described honestly, and its limits must be stated. A convenience sample, drawn from whoever was available, can still be useful for exploration, but it cannot support general claims about a wider population. Stating the method lets readers judge how far the conclusions can travel. Claiming general conclusions from a convenience sample, without acknowledging the method, is a form of overstatement that careful analysts avoid.

Sample size matters, but bigger is not automatically better if the sample is biased. A large biased sample gives a precise estimate of the wrong thing. Size matters because small samples are noisy: a few unusual records can swing averages and percentages dramatically. The size needed depends on the question, the variability in the data, and the precision required. As a practical rule, report the sample size alongside every finding, and be especially cautious with findings from small samples or small subgroups within a sample.

Documentation completes the sampling story. Record how the population was defined, how the sample was selected, when it was selected, and how large it is. If some selected members did not respond, record the response rate and consider whether non-respondents might differ from respondents. A survey with a low response rate may describe only those motivated enough to reply, not the whole population. That limitation should appear in the report, not in a footnote that readers will miss.

Sampling also connects to the earlier topics of bias and coverage. A sample that excludes a group, whether by design or by accident, inherits the bias of that exclusion. Checking who is missing from a sample, comparing the sample's composition with known features of the population where possible, and reporting discrepancies are all part of honest sampling. A sample is a lens on a population. Its value depends on how clearly you can describe the lens, its size, and its blind spots.

Check: Why should samples be random where possible, documented, and large enough for their conclusions?

Section 30: Bias in Data

Bias in data means a systematic distortion that pushes results in a particular direction, away from the truth about the population or phenomenon being studied. Bias differs from random noise. Noise varies unpredictably and tends to average out over many records. Bias pulls consistently one way, so collecting more data of the same biased kind does not fix it. It simply produces a more precise estimate of a distorted picture. Recognising bias is therefore essential, because it determines whether more data helps or merely reinforces an error.

Bias has many causes. Selection bias occurs when the records in a dataset do not represent the whole population, because some groups were more likely to be included than others. Measurement bias occurs when a quantity is measured inconsistently or inaccurately, so that values systematically overstate or understate the truth. A faulty scale, a leading survey question, or a definition that changed over time can all create measurement bias. Historical bias occurs when past patterns embedded in data reflect past conditions or past decisions, and those patterns are then treated as if they described the present fairly.

Missing data can also create bias, when the missing records differ systematically from the records present. If dissatisfied customers are less likely to complete a survey, survey results will overstate satisfaction. If a sensor fails during extreme conditions, recorded data will understate extremes. In each case, the bias arises not from the data that exists, but from the data that does not exist and from how it differs from the data that does.

Suspected bias must be reported with its likely effect on conclusions. It is not enough to note vaguely that bias may exist. A useful report says which kind of bias is suspected, why it is suspected, and in which direction it likely pushes the results. For example, if a survey under-represents one region, the report should say that conclusions about that region are weaker, and that overall results may tilt toward the over-represented regions. That specificity lets decision makers judge how much weight to place on the findings, and whether additional data collection is needed.

Bias cannot always be removed, but it can often be reduced. Improving selection, fixing measurement, collecting data from missing groups, or analysing groups separately can all help. When bias cannot be removed, the honest response is to limit conclusions to what the biased data can support, and to state that limit clearly. A conclusion restricted to the group actually observed is more trustworthy than a sweeping conclusion that pretends the data represented everyone.

Finally, bias has an ethical dimension when data concerns people. Decisions based on biased data can unfairly disadvantage groups that were under-represented or mismeasured. Analysts have a responsibility to look for bias, especially when findings will affect people, and to raise concerns even when doing so complicates a neat story. A dataset is not neutral merely because it is numeric. Its biases reflect how it was collected, from whom, and with what tools, and a good analyst makes those influences visible.

Check: What causes bias in data, and why must suspected bias be reported with its likely effect on conclusions?

Section 31: Data Quality in Concept

Data quality describes how fit a dataset is for its intended analysis. A dataset can be large and still be low quality, if its values are inaccurate, incomplete, inconsistent, or out of date. Quality is not a single property. It has several dimensions, and checking each one separately gives a clearer picture than a vague judgement that data is good or bad. The main dimensions are accuracy, completeness, consistency, timeliness, and validity, and each deserves a brief examination before analysis begins.

Accuracy asks whether values reflect reality. A price recorded incorrectly, a date typed wrongly, or a quantity miscounted are accuracy problems. Completeness asks whether all expected records and fields are present. Missing days, missing regions, or fields blank for many records are completeness problems. Consistency asks whether values agree across the dataset and across sources. A category spelled several ways, or a total that does not match the sum of its parts, are consistency problems. Timeliness asks whether data is current enough for the question. Last year's prices may not describe this year's market. Validity asks whether values conform to expected formats and ranges, such as dates that are real calendar dates and quantities that are not negative where negativity makes no sense.

Quality problems must be reported honestly, not hidden by cleaning. Cleaning can fix some problems, such as standardising spellings, but it cannot fix all of them, and it should not be used to disguise problems that remain. If a field is missing for forty percent of records, the report should say so, even after a handling decision has been made. If accuracy is doubtful for a particular source, the report should say so, and conclusions that depend on that source should be qualified. Hiding quality problems makes results look stronger than they are, and it prevents decision makers from seeking better data.

Quality checks should be systematic. For each dimension, perform a simple check and record the result. Count missing values for completeness. Check formats and ranges for validity. Compare totals with their parts for consistency. Compare dates with the period the data should cover for timeliness. Sample values against source records for accuracy where possible. Those checks do not require advanced tools. They require patience and a willingness to look at the data before trusting it.

Quality also affects which questions a dataset can answer. A dataset with poor completeness for one region cannot support strong conclusions about that region, however good its data for other regions may be. A dataset with doubtful accuracy for a key field cannot support precise claims about that field, though it might still support broader patterns. Matching the strength of conclusions to the quality of the data is a central skill of analysis, and it is more honest than forcing every question to yield a confident answer.

Finally, quality is a property of a dataset for a purpose, not an absolute grade. Data that is good enough for exploring a rough trend may not be good enough for a financial decision. Stating the purpose alongside the quality assessment lets readers judge fitness for that purpose. A quality report that lists dimensions, findings, and remaining problems is a valuable part of any analysis, because it tells readers exactly how far they should trust the numbers that follow.

Check: What dimensions make up data quality, and why must quality problems be reported honestly?

Section 32: Reproducibility in Analysis

Reproducibility means that another person, or your future self, can repeat an analysis from the same data and the same steps and obtain the same results. It is the foundation of trust in analysis. A result that cannot be reproduced cannot be checked, and a result that cannot be checked should not guide important decisions. Reproducibility is not an extra polish for finished work. It is a working habit that shapes how analysis is done from the start.

Several practices make analysis reproducible. First, preserve the original data unchanged, and work on copies. Second, record every step, from loading and cleaning through grouping, aggregation, and charting, in an ordered script or notebook that can be run again. Third, document decisions, such as how missing values were handled, how duplicates were defined, and how outliers were treated, so that those decisions are part of the record, not hidden in memory. Fourth, record the tools and their versions where relevant, because different versions can sometimes produce different results. Fifth, keep the code, the data, and the report together, so that a reviewer can trace a number in the report back to the records and steps that produced it.

Testing reproducibility means actually re-running the analysis from scratch. Delete intermediate outputs, start from the original data, run the steps in order, and compare the new results with the reported results. If they differ, something was done by hand, out of order, or with a setting that was not recorded. That difference is valuable information, not an embarrassment. It reveals a gap in the record that should be fixed before the analysis is shared. Analysts who test reproducibility before sharing catch errors that would otherwise reach decision makers.

Order matters in this test, which is why analysis steps should run from top to bottom without relying on earlier manual actions. A notebook that only works if cells are run in a particular scattered order is not reproducible in practice, even if its author knows the order. A script that depends on a file edited by hand, without recording the edit, is similarly fragile. The test of re-running from scratch exposes these dependencies, because a fresh run has no memory of manual steps.

Reproducibility also helps the original analyst. Analyses are often revisited months later, when a question returns or new data arrives. An analysis whose steps are recorded can be updated quickly and confidently. An analysis whose steps live only in memory must be rebuilt, with a high risk of producing different results for the same question. The time spent documenting steps is repaid the first time an analysis needs to be repeated.

Finally, reproducibility supports honesty. When steps are visible, reviewers can challenge a cleaning decision, test an alternative grouping, or check the effect of an outlier. That scrutiny strengthens good analysis and exposes weak analysis, which is exactly what a decision maker needs. A reproducible analysis invites checking. An irreproducible analysis asks for trust without offering a way to verify it, and careful readers should be wary of that bargain.

Check: What does reproducibility require, and why test it by re-running analysis from scratch?

Section 33: Notebooks in Concept

An analysis notebook is a document that combines code, results, and explanatory writing in a single file, arranged in cells that can be run in order. A notebook lets an analyst load data, compute summaries, draw charts, and explain each step in prose, all in one place. Readers can follow the reasoning, see the outputs beside the code that produced them, and, if they have the data and tools, run the notebook themselves to verify the results. Notebooks are therefore a popular format for exploratory analysis, teaching, and sharing work in progress.

Notebooks serve several purposes. They record the analysis as it was performed, rather than as a polished summary written afterwards. They make intermediate results visible, so that a surprising summary can be traced back to the step that produced it. They also support iteration, because an analyst can change a filter, rerun a cell, and see the new result immediately. That speed is valuable during exploration, when questions are still being refined and many approaches are being tried.

Notebooks also carry a well-known risk: cells can be run out of order. An analyst might run a later cell, change an earlier cell, and then share the notebook with outputs that no longer match the code in order. A reader who runs the notebook from top to bottom might then obtain different results from the ones displayed. That mismatch undermines trust, because the displayed results cannot be reproduced by following the notebook as written. It can also hide errors, when a variable left over from an earlier experiment affects a later calculation in a way that a fresh run would not reproduce.

The remedy is simple and strict. Before sharing a notebook, restart it and run all cells from top to bottom, in order, without skipping or reordering. Then check that the outputs match the story told in the prose. If a cell fails, or if an output changes, fix the notebook before sharing it. That fresh run is the notebook equivalent of testing reproducibility from scratch, and it should be treated as a required final step, not an optional tidy-up.

Good notebook habits go beyond the fresh run. Keep notebooks focused on one question or one analysis, rather than accumulating unrelated experiments in a single file. Write prose that explains why each step is taken, not only what it does. A cell that filters data should be accompanied by a sentence stating the filter and its reason. Remove or clearly label experimental cells that did not lead anywhere, so readers are not confused by dead ends. Keep the data source and any cleaning decisions documented in the notebook itself, so the file stands alone as a record.

Notebooks are tools for thinking and for communication, and their strength is that code and explanation sit together. That strength becomes a weakness when order is neglected. A notebook run top to bottom, with clear prose and documented decisions, is a trustworthy record of an analysis. A notebook shared with stale, out-of-order outputs is a puzzle that readers must solve before they can trust anything in it. The difference is a single habit: restart, run all, check, and only then share.

Check: What are analysis notebooks for, and why must they be run top to bottom before sharing?

Section 34: Data Libraries in Concept

Data libraries are software packages that provide ready-made tools for working with tables, numbers, and charts. In concept, a table library provides a table structure with operations for filtering, grouping, sorting, aggregating, and joining, of the kinds described in this chapter. A numeric library provides efficient operations on lists of numbers, such as sums, averages, and other summaries. A charting library provides ways to turn tables and summaries into labelled charts. Together, these kinds of libraries let analysts work with large datasets more quickly than plain code alone would allow.

Libraries help in several ways. They handle common operations reliably, so analysts do not need to write their own code for grouping or sorting from scratch on every project. They are tested by many users, which means common errors are more likely to have been found and fixed. They also provide a shared vocabulary, so that analysts working on the same kind of problem can describe their steps in terms others recognise.

This chapter is deliberately conceptual about libraries, and that choice deserves explanation. Library details change over time. Function names are added, changed, or retired. Default behaviours shift. A chapter that taught specific function signatures in detail would become outdated, and learners who memorised those details might be misled when they encounter a newer version. Concepts, by contrast, remain stable. Filtering means the same thing in every tool. Grouping, aggregation, joining, and charting are ideas that transfer across libraries and across languages. Learning the concepts first means you can learn any particular library more quickly, because you already know what you are looking for.

Conceptual treatment also prevents a common beginner error, which is to confuse knowing a library with understanding analysis. A person can produce a grouped summary with a library call without understanding what grouping means, why group sizes matter, or how the result should be checked. That person can generate impressive output that is nevertheless wrong, because a misspelled category split a group, or a filter was applied in the wrong order. The concepts in this chapter, from cleaning and missing values to bias and reproducibility, are what let you judge whether library output is trustworthy. The library performs the operation. You remain responsible for the reasoning.

When you do begin using a particular library, approach it with the habits of this chapter. Read its current documentation for the operation you need, rather than relying on memory from an older version. Check row counts and column names after loading. Verify grouped totals against overall totals. Chart distributions before choosing summaries. Document the steps you took, in whatever form the library supports. Those habits matter more than fluency with any single package, because they protect every analysis, in every tool, from the same quiet errors.

Libraries are powerful assistants, not substitutes for judgement. They speed up the mechanical parts of analysis, leaving more time for the parts that require thought: asking a good question, checking data quality, choosing appropriate summaries, and reporting honestly. Use them for that purpose, learn their current details from their own documentation when you need them, and let the concepts in this chapter guide how you use them.

Check: What do data libraries provide in concept, and why is this chapter deliberately conceptual about them?

Section 35: Analysis Workflow

An analysis workflow is an ordered sequence of steps that takes you from a question to a report. Following a workflow keeps analysis systematic, prevents steps from being skipped, and makes the work easier to check and repeat. Different analysts describe the steps in slightly different words, but the order is broadly consistent, and the discipline of following it is more important than the exact labels used.

The workflow begins with the question. Write down a specific question and decide what would count as an answer, before examining data. Next, understand the dataset: its source, its period, its coverage, its fields, and its kinds. Then load the data and verify row counts, column names, and sample values. Then clean the data, fixing inconsistencies, handling missing values and duplicates, investigating outliers, and documenting every change. Cleaning is placed before analysis deliberately, because summaries computed on dirty data inherit its problems.

Exploration follows cleaning. Compute descriptive statistics, chart distributions, group the data by relevant fields, and look for patterns, differences, and surprises. Exploration is open-ended, but it should still be guided by the original question, so that it produces relevant insight rather than a collection of unrelated curiosities. When exploration suggests a more precise question, or a hypothesis worth testing, write that new question down as well, and note that it arose during exploration. That note matters, because a question formed after seeing data carries different weight from a question fixed beforehand.

Analysis proper then tests the question against the cleaned, explored data. Compute the summaries, comparisons, or correlations that the question requires, check them with the verification habits of this chapter, and consider alternative explanations. Grouped results should be checked against overall results. Correlations should be checked for outliers and third variables. Trends should be checked for gaps and measurement changes. This checking stage is where many errors are caught, and it should never be rushed.

Reporting is the final step. A report states the question, describes the data and its limitations, presents findings with honest charts, states uncertainty, and suggests what to investigate next. Reporting is not an afterthought. It is the point of the whole workflow, because analysis that is never communicated clearly cannot inform decisions. A report that omits limitations or uncertainty may look stronger, but it is less trustworthy than a report that states them plainly.

Question changes deserve special attention throughout the workflow. Questions legitimately evolve as you learn about the data. A dataset may lack a field you assumed it had, or exploration may reveal a more interesting question than the one you started with. Changing a question is acceptable, provided the change is documented, with the reason and the point in the workflow at which it occurred. Undocumented question changes, especially changes made after seeing results, undermine trust, because they allow the question to be fitted to pleasing results. A workflow with documented changes is honest about its own history, and that honesty is what makes its conclusions worth acting on.

Check: What are the ordered steps of an analysis workflow, and why must question changes be documented?

Section 36: Exploratory Analysis

Exploratory analysis is the open-ended stage of analysis in which you examine a dataset to discover its patterns, quirks, and possibilities, before committing to a specific test or conclusion. It involves computing descriptive statistics, charting distributions, grouping by interesting fields, looking at correlations, and following up on surprises. Exploration is where you get to know the data, in the same way that reading a book's contents page and first chapters helps you understand what it contains before you study it in detail.

Exploration is valuable because datasets often contain features that a fixed question would miss. A distribution might be multi-peaked, suggesting distinct groups. A category might dominate unexpectedly. A trend might reverse within subgroups. Discovering those features early shapes better questions and prevents wasted effort on analyses that assume a simpler structure than the data actually has. Exploration also reveals data problems, such as unexpected missing values or implausible extremes, that should be addressed before formal analysis begins.

Exploration must be guided, but lightly. The original question provides direction, so exploration does not become aimless wandering. Within that direction, curiosity is welcome. Try grouping by a field you had not considered. Chart a distribution you had not planned to chart. Look at a correlation that exploration suggests. The goal is to learn what the data can and cannot support, and to generate ideas worth testing more rigorously. Exploration is a time for questions, not for verdicts.

That distinction is crucial for reporting. Findings from exploration must be reported as suggestions, not as confirmations. A pattern found by exploring many possibilities might have arisen by chance, because exploring many comparisons increases the chance that some comparison will look striking even when nothing real is happening. An exploratory pattern is therefore a hypothesis to be tested on fresh data, or with a more rigorous method, not a conclusion to be acted on immediately. Reporting it as a suggestion, with the exploration process described, lets readers judge its weight correctly.

Documentation matters in exploration as much as in formal analysis. Note which explorations you tried, what you found, and which findings you decided to pursue. That record prevents a subtle form of self-deception, in which an analyst remembers only the striking patterns and forgets the many uninteresting ones that were also examined. It also lets another analyst see how a hypothesis arose, and whether it was formed before or after the data was examined.

Exploratory analysis is sometimes dismissed as mere playing with data, but that dismissal misses its purpose. Good exploration is systematic, documented, and honest about its own status. It maps the terrain before a route is chosen, reveals problems before they corrupt conclusions, and generates the questions that formal analysis will answer. Treat its findings with appropriate caution, as promising leads rather than proven facts, and exploration becomes one of the most productive stages of the whole workflow.

Check: What is exploratory analysis, and why must its findings be reported as suggestions, not confirmations?

Section 37: Forming a Hypothesis

A hypothesis is a specific, testable statement about what you expect to find in data. In analysis, forming a hypothesis means turning a general question into a precise prediction that data can support or contradict. For example, a general question about whether a promotion increased sales becomes a hypothesis when it states which product, which period, which measure, and what size of change would count as an increase. That precision is what makes a hypothesis useful, because it turns a vague curiosity into something that can be checked.

A useful hypothesis has several qualities. It is specific, naming the variables, the groups, and the time period involved. It is testable, meaning the data needed to check it can actually be obtained and the result can be observed. It is falsifiable, meaning it could turn out to be wrong, so that confirming it carries real information. A statement that cannot be wrong, because it is so vague that any result fits it, teaches nothing. A statement that predicts a direction, such as sales in the promoted category rising compared with the previous month, can be contradicted by the data, and that risk is what gives it value.

Writing a hypothesis down before testing it is essential. When a hypothesis is written beforehand, the test is a genuine check, because the prediction was fixed before the result was known. When a hypothesis is written after seeing the result, it is merely a description of what happened, dressed as a prediction. The difference matters for trust. A pre-written hypothesis that turns out correct is evidence. A post-hoc hypothesis that matches the data is not evidence, because it was chosen to fit the data it claims to predict.

Pre-writing also clarifies what counts as support. If the hypothesis predicts a rise of a certain size over a certain period, the test can be judged against that standard, rather than against a shifting standard invented after the result is known. A small rise that falls short of the predicted size should be reported as falling short, not reinterpreted as support by lowering the bar. That discipline protects analysts from fooling themselves, and it protects readers from being misled by selective reporting.

Hypotheses can arise at different stages. Some are formed before any data is examined, from theory, experience, or a business question. Others arise during exploration, when a pattern suggests a prediction worth testing. Exploratory hypotheses are legitimate, provided they are labelled as exploratory and tested on fresh data or with a method that accounts for their origin. The key is transparency about timing. Readers should always be able to tell whether a hypothesis preceded the test or followed the observation.

Finally, a hypothesis that turns out wrong is not a failure. It is information. It rules out one explanation and directs attention to others. Analysis progresses by testing specific predictions, learning from those that fail, and refining the next prediction in light of what was learned. An analyst who only reports successful hypotheses, hiding the ones that failed, gives a distorted picture of how much was tested and how surprising the successes really were. Record all hypotheses, report their outcomes honestly, and let the pattern of successes and failures speak for itself.

Check: What makes a hypothesis useful, and why write it down before testing it?

Section 38: Reporting Findings

Reporting findings is the final stage of analysis, and it determines whether good work actually informs decisions. A report that is unclear, incomplete, or overstated wastes the analysis behind it, because readers cannot tell what was asked, what was found, or how much to trust it. A good report follows a clear structure, states uncertainty honestly, and lets readers trace conclusions back to evidence.

A sensible structure begins with the question. State the specific question the analysis set out to answer, and what would count as an answer. Next, describe the data: its source, its period, its coverage, its size, and its limitations. Readers need this context before they can judge findings. If data covered only one region or one month, say so at the start, not in a footnote. If important fields were missing, say so plainly, with counts.

The findings section follows, presenting results in a logical order, usually from overall summaries to grouped or detailed results. Each finding should be stated in plain language, supported by a number or a chart, and accompanied by the context needed to interpret it, such as record counts, group sizes, and spread alongside averages. Charts should be honestly labelled, with titles, axes, and units, as discussed earlier in this chapter. Findings should be separated from interpretation, so readers can see what the data shows and what the analyst thinks it means.

Uncertainty must be stated honestly throughout. No analysis is free of uncertainty. Samples vary, measurements err, data goes missing, and assumptions shape results. A report should name the main sources of uncertainty and explain how they affect conclusions. Does a small sample make a finding tentative? Does a suspected bias push results in a particular direction? Would a different handling of missing values change the conclusion? Answering those questions does not weaken a report. It strengthens it, because readers can see that the analyst has considered what could be wrong.

Limitations and next steps close the report. Limitations summarise what the analysis could not do, whether because of data gaps, a short period, or a method that cannot establish causation. Next steps suggest what should be investigated further, such as collecting data from a missing group, testing a hypothesis on fresh data, or examining a subgroup in more detail. A report that ends with limitations and next steps is not admitting defeat. It is showing the boundary of current knowledge and pointing toward how that boundary can be pushed further.

Finally, write for the reader, not for yourself. Avoid jargon where plain language will do. Define terms on first use. Keep sentences clear and paragraphs focused. A report is a communication tool, and its success is measured by whether readers understand the question, the evidence, the uncertainty, and the conclusion, and can act wisely on that understanding. Clarity and honesty are not stylistic preferences in reporting. They are the qualities that make analysis useful.

Check: What structure should an analysis report follow, and why must uncertainty be stated honestly?

Section 39: Ethics and Privacy in Analysis

Analysing data about people carries ethical duties that go beyond technical correctness. Data about people can affect their opportunities, their privacy, and their dignity, and analysts who handle such data must consider those effects as part of their work, not as an afterthought. Ethics in analysis is not a separate topic to be added at the end. It shapes which data is collected, how it is handled, how findings are reported, and who sees them.

Privacy is the first duty. Collect only the data needed to answer the question, and avoid sensitive fields that the analysis does not require. When personal identifiers are not needed, remove them before analysis, and work with aggregated or de-identified data where possible. Even de-identified data can sometimes identify people, especially in small datasets or when distinctive combinations of fields are present, so consider identification risk before sharing results or examples. Text quotations deserve particular care, because a distinctive phrase can reveal its author even when a name is removed.

Consent and purpose also matter. Data collected for one purpose should not be reused for a very different purpose without considering whether that reuse is fair to the people it concerns. A dataset of customer feedback collected to improve a service is not automatically suitable for public reporting that identifies individuals. When reuse is contemplated, check the original purpose, any promises made to the people involved, and any rules that govern the data. When in doubt, seek guidance rather than assuming permission.

Fairness is a further duty, closely linked to bias. Findings that will affect groups of people should be checked for bias in selection, measurement, and missing data, as discussed earlier in this chapter. A finding that disadvantages a group because that group was under-represented or mismeasured is not merely a technical error. It is an unfair outcome built on flawed evidence. Analysts should raise bias concerns even when doing so complicates a neat story or delays a decision, because the cost of acting on biased findings can fall on people who had no voice in the analysis.

Sharing findings requires care as well. Reports should be shared with those who need them for the decision at hand, and not more widely than necessary when they contain sensitive information. Aggregated findings are usually safer to share than record-level data. When findings are shared publicly, they should be aggregated enough that individuals cannot be identified, and they should include limitations and uncertainty, so that public readers are not misled by a simplified headline. Selective sharing, in which only favourable findings are released, is a form of dishonesty, because it gives a partial picture while claiming to present evidence.

Finally, analysts should consider the consequences of acting on their findings. A correlation presented as a cause, a small sample presented as a general truth, or a biased dataset presented as neutral can all lead to decisions that harm people. Thinking through consequences before reporting, and stating limitations clearly, are practical ways to exercise ethical responsibility. Good analysis is not only accurate. It is careful about the people behind the numbers, and about the effects its conclusions will have on them.

Check: What ethical duties apply when analysing data about people, and how should findings be shared?

Section 40: Common Analysis Pitfalls

Common analysis pitfalls are recurring errors that trap even experienced analysts, because they arise from habits that feel natural but lead to wrong conclusions. Learning to recognise them is as important as learning the techniques themselves, because a technique applied without awareness of its pitfalls can produce confident errors. This section collects the pitfalls that appear throughout this chapter, so you can guard against them systematically.

One pitfall is starting without a question. Aimless exploration produces patterns without answers, and invites the analyst to retrofit a question to a pleasing result. Another is trusting data without checking it. Skipping loading verification, cleaning documentation, or quality checks allows errors, duplicates, and missing values to flow silently into summaries and charts. A third pitfall is reporting averages without spread, group sizes, or context, which makes varied, small, or skewed data look more solid than it is.

Confusing correlation with causation is a further pitfall, with real consequences when decisions follow. So is removing outliers automatically, filling missing values without justification, or treating empty pivot cells as zeros. Each of those actions hides a judgement behind a technical step, and each can distort results in a direction that favours a preferred story. The antidote in every case is the same: investigate first, decide for a stated reason, and document the decision so it can be checked.

Visual pitfalls deserve their own attention. Truncated axes exaggerate differences. Cherry-picked periods suggest trends that longer histories contradict. Unlabelled charts invite misreading. Charts copied without their context lose the qualifications that made them honest. A chart that looks dramatic should prompt a second look at its axes, its period, and its source data, before its message is accepted.

Checklists guard against these pitfalls precisely because pitfalls are easy to overlook under time pressure. A checklist for an analysis might include: question written down, data source and period recorded, row counts verified, cleaning documented, missing values counted, duplicates checked, outliers investigated, spread reported with averages, group sizes reported, charts labelled, limitations stated, and reproducibility tested. Running through that list before sharing a report takes little time and catches errors that familiarity with the work makes invisible.

Checklists work best when they are used consistently, not only when an analysis feels risky. The analyses that feel safest are often the ones where a familiar dataset and a routine method lull the analyst into skipping checks. Making the checklist a habit, applied to every analysis regardless of size, builds a safety net that catches the occasional slip before it reaches a decision maker.

Finally, treat pitfalls as learning opportunities rather than as signs of incompetence. Every analyst encounters them. The difference between a careless analyst and a careful one is not that the careful analyst never slips, but that the careful analyst has systems, such as documentation, verification, and checklists, that catch slips early. Build those systems now, while the habits are forming, and they will protect your work throughout your career.

Check: What common pitfalls must analysts avoid, and how do checklists guard against them?

Section 41: Analysis with CSV and JSON Data

Comma-separated and JSON data are two common file formats, and analysis differs between them mainly in how data is loaded and shaped, while the overall workflow stays the same. Understanding both formats, and what stays constant across them, prepares you to work with real datasets, which rarely arrive in a single convenient form.

Comma-separated data, often called CSV, stores tabular data as plain text, with one row per line and fields separated by commas. Its structure matches the rows-and-columns model directly, which makes loading straightforward. A CSV file usually begins with a header row naming the columns, followed by one record per line. Commas inside a field, line breaks inside a field, and inconsistent quoting can all confuse a naive reader. A robust loading approach handles quoting correctly, rather than splitting lines on commas by hand. CSV files also lack built-in information about field kinds, so numbers and dates must be checked and converted after loading, as discussed earlier.

JSON data stores records as structured text, with named fields and nested structures. A JSON dataset might hold a list of records, each with fields, some of which contain lists or nested objects of their own. That flexibility suits data that is naturally hierarchical, such as an order containing a list of items, or a user record containing an address object. The challenge for tabular analysis is flattening. Nested structures must be unpacked into rows and columns before grouping, aggregation, or charting can proceed in the usual way. Flattening involves choices, such as how to handle a list of items within one record, and those choices should be documented, because different flattening rules produce different row counts and totals.

Despite these loading differences, the workflow after loading is the same. Whether data arrived as CSV or JSON, you must verify row counts and column names, check field kinds against actual values, clean inconsistencies, count missing values, check duplicates, investigate outliers, compute descriptive statistics, group and aggregate, chart honestly, and report with limitations. The format changes the first steps. It does not change the standards for trust. An analysis of JSON data that skips cleaning is no more reliable than an analysis of CSV data that skips it.

Format-specific checks are worth adding to the general checklist. For CSV, check for extra header rows, inconsistent column counts across lines, and encoding problems that garble text. For JSON, check for records with missing fields, fields that sometimes hold a single value and sometimes a list, and nested structures that flatten unexpectedly. In both cases, compare the loaded table with the source: does the row count match the number of records in the file, and do sample values match the source values?

A practical approach is to write a small loading script that reads the file, reports row counts and column names, shows sample rows, and saves the loaded table for the rest of the analysis. That script becomes part of the reproducible record, so the loading step can be repeated exactly. Whether the source is CSV or JSON, that discipline turns a format difference into a documented step, rather than a hidden source of error. Formats will continue to vary, but a workflow that verifies, cleans, and documents after loading will serve you for any format you meet.

Check: How does analysis differ for CSV versus JSON data, and what stays the same in the workflow?

Section 42: Spreadsheets Versus Code Analysis

Spreadsheets and code analysis are two ways of working with data, and each suits different situations. A spreadsheet is a grid of cells with formulas, sorting, filtering, and charting built in, designed for interactive work that a person can see and adjust directly. Code analysis means writing a program that loads data, performs steps in order, and produces summaries and charts, with every step recorded as text that can be run again. Both approaches can produce correct results. The choice depends on the size, repetition, and audit needs of the task.

Spreadsheets suit small, one-off analyses where interactivity matters. When a dataset is small enough to inspect by eye, and the analysis will be done once, a spreadsheet lets you explore quickly, try a formula, see the result immediately, and adjust. Spreadsheets are also accessible to people who do not program, which makes them useful for sharing a simple summary with colleagues who can open the file and see both values and formulas. For a quick check, a small table, or a single report, a spreadsheet is often the fastest honest route.

Code analysis is preferred when several conditions apply. When a dataset is large, code handles it more reliably than manual cell operations, which become error-prone as rows multiply. When an analysis must be repeated, for example weekly or monthly on new data, code can be rerun on the new file with the same steps, while a spreadsheet would require repeating manual steps each time, with a risk of inconsistency. When reproducibility matters, code provides a complete record of steps that a reviewer can run from scratch, while spreadsheet steps may be scattered across cells, menus, and manual edits that are hard to reconstruct.

Auditability is a further difference. In code, the sequence of steps is visible in order, and changes are tracked as text. In a spreadsheet, a value in a cell may result from a formula, a pasted value, or a manual edit, and distinguishing those cases requires inspecting each cell. That opacity is not a flaw for a quick personal check, but it becomes a risk when results inform decisions or when another person must verify the work. Code does not eliminate errors, but it makes them easier to find, because the steps that produced a result are explicit.

A hybrid approach is common in practice. An analyst might explore data in a spreadsheet to understand its shape, then implement the final analysis in code for repetition and audit. Or code might produce a summary table that is then opened in a spreadsheet for presentation. The key is to know which tool is doing which job, and to keep the authoritative steps in the form that can be repeated and checked.

Choosing between them is therefore a judgement about the task, not a loyalty to a tool. Ask how large the data is, how often the analysis will repeat, who must verify it, and how much manual handling it would require. If the answers point to a small, single, visible task, a spreadsheet may serve well. If they point to a large, repeated, or auditable task, code is the safer choice. Either way, the standards of this chapter still apply: a clear question, verified loading, documented cleaning, honest summaries, labelled charts, and stated limitations.

Check: When are spreadsheets suitable, and when should code analysis be preferred instead?

Section 43: Automating Repeated Analysis

Automating repeated analysis means writing a program that performs the same analysis steps on new data, on a schedule or on demand, without manual repetition each time. Many analyses repeat regularly: weekly sales summaries, monthly activity reports, daily quality checks, and similar tasks. Performing those steps by hand each time is slow, tedious, and error-prone, because manual steps vary from run to run. Automation replaces that variation with a consistent, repeatable process.

The benefits of automation go beyond speed. A scripted analysis applies the same cleaning rules, the same filters, and the same aggregations every time, so results from different weeks are comparable. A manual process might apply a filter slightly differently one week, or forget a cleaning step, producing a change that looks like a real trend but is really a procedural difference. Automation also creates a record. The script itself documents the steps, so a reviewer can see exactly what was done, and a new analyst can take over the task without reconstructing it from memory.

Automation must still include checks, because new data can break assumptions that held for old data. A script that worked last month may fail, or worse, produce wrong results silently, when a new file has a different column name, a new category value, an unexpected missing field, or a format change. Automated runs should therefore verify row counts against expected ranges, check that expected columns are present, count missing values, and flag unexpected values for human review. Those checks turn automation from a blind machine into a supervised process that raises an alarm when data changes.

Failures in automated runs should be visible, not silent. If a check fails, the run should report the failure clearly, rather than producing a partial report that looks complete. If a chart cannot be produced because data is missing, the report should say so, rather than showing an old chart as if it were current. Designing for visible failure is a form of honesty in automation. It ensures that decision makers are not misled by stale or incomplete output that automation made easy to produce.

Human review remains part of automated analysis, especially for conclusions. Automation can compute summaries and draw charts, but interpreting them, checking for bias, considering third variables, and deciding what to report still require judgement. A sensible pattern is for automation to produce a draft report with checks and flags, which a person reviews before it is shared. That pattern combines the consistency of automation with the judgement of a human analyst, and it avoids both manual drudgery and unreviewed machine output.

Finally, automated analysis should be tested like any program. Run it on known data and confirm it produces expected results. Run it on data with deliberate problems, such as missing values or new categories, and confirm it flags them rather than hiding them. Keep the script under version control, so changes to the analysis are tracked and can be reversed. Automation is not a shortcut around the standards of this chapter. It is a way of applying those standards consistently, every time, provided its checks and its human review are kept in place.

Check: Why automate repeated analysis, and what checks must automated runs still include?

Section 44: Honest Visualisation

An honest visualisation is a chart or diagram that shows data accurately, completely enough to support fair interpretation, and without tricks that exaggerate, hide, or distort the underlying pattern. Honesty in visualisation is not only about correct numbers. A chart can use correct numbers and still mislead, through its axes, its scale, its selection of data, or its emphasis. Building honest charts is therefore a core skill, not a matter of polish.

Several practices make a visualisation honest. Axes should start at zero for bar charts, so that bar lengths represent values proportionally. Scales should be consistent across charts that are meant to be compared, so that a reader is not tricked by different ranges. Titles, axis labels, and units should state exactly what is shown, including the period covered and any filtering applied. Data sources should be noted, so readers know where the numbers came from. Uncertainty, such as small sample sizes or wide spread, should be visible or noted, rather than hidden behind a clean-looking average.

Selective practices must be avoided. Cherry-picking a time period that shows a favourable trend, while omitting the longer history, is selective and misleading, even if every value shown is correct. Omitting a relevant group, such as a poorly performing category, while showing the others, gives a partial picture that claims to be complete. Highlighting a single point without showing its context, such as an average without its spread, similarly distorts interpretation. Each selective choice shapes the story the chart tells, and readers who do not know what was omitted cannot correct for it.

Colour and design also carry honesty implications. Colour should distinguish categories neutrally, not imply that one category is good and another bad unless the data supports that judgement. Three-dimensional effects, decorative icons, and distorted shapes make values harder to read accurately, and they should be avoided when precision matters. Simplicity is usually more honest than decoration, because it leaves less room for visual tricks to hide.

A useful test for any visualisation is to ask what a stranger would conclude from the chart alone, without the surrounding text. Would they grasp the main pattern correctly? Would they notice the period, the population, and the scale? Would they be misled about the size of differences? If the chart alone suggests a stronger, weaker, or different story from the data, revise it before sharing. Charts are often shared on their own, separated from their reports, so a chart that depends on surrounding paragraphs to be honest is a fragile chart.

Honest visualisation also means showing what is missing. Gaps in a time series should appear as gaps. Small groups should be flagged. Empty categories should be noted rather than silently dropped. Those choices may make a chart look less tidy, but tidiness bought by hiding gaps is a false tidiness. A chart that shows its own limits invites trust, because readers can see that nothing important has been concealed. That trust is the real purpose of visualisation: not to impress, but to let readers see the evidence clearly and judge it for themselves.

Check: What makes a visualisation honest, and what selective practices must it avoid?

Section 45: Storytelling with Data

Data storytelling means guiding readers through findings in a clear sequence, so they understand not only what the numbers are, but what they mean and what should happen next. A story in this sense is not fiction. It is a structured presentation: a question, evidence, interpretation, and a conclusion, arranged so that each step follows from the last. Without that structure, a report becomes a pile of charts and tables that readers must assemble for themselves, and many will assemble them incorrectly or not at all.

A good data story begins with the question and why it matters. Readers need to know what decision or curiosity motivated the analysis, before they are shown numbers. Next, it presents evidence in an order that builds understanding, usually from overall patterns to specific details. A story might start with total sales over time, then show the breakdown by category, then highlight the category that drove a change. That order helps readers follow the reasoning, because each finding prepares them for the next one, rather than jumping between unrelated results.

Interpretation follows evidence, and the two should be kept distinct. Evidence states what the data shows, such as totals rising by a certain amount over a stated period. Interpretation suggests what that evidence might mean, such as a promotion coinciding with the rise. Keeping them separate lets readers agree with the evidence while disagreeing with the interpretation, which is a healthy and normal part of reading analysis. A story that blends evidence and interpretation into a single claim makes disagreement harder and hides the analyst's judgement behind the numbers.

Honesty about limitations is part of storytelling, not an interruption to it. A story that acknowledges a small sample, a missing group, a gap in the data, or an alternative explanation is more persuasive, not less, because readers can see that the analyst has considered weaknesses. Limitations should appear where they affect interpretation, not only in a final paragraph that readers may skip. If a finding rests on a filtered subset, say so when the finding is presented. If a correlation cannot establish causation, say so when the correlation is discussed.

Endings matter in data stories. A story should close with a clear conclusion that answers the original question, states how confident the analyst is, and suggests what to investigate next. A vague ending, such as data shows interesting patterns, leaves readers unsure what to do. A precise ending, such as sales in one category rose during the promotion period, but the data cannot show whether the promotion caused the rise, and a controlled comparison is the next step, gives readers both an answer and a direction.

Finally, remember that storytelling is a responsibility. The order in which findings are presented, the findings chosen for emphasis, and the language used to describe them all shape how readers understand the data. Choosing an order that reflects the evidence, emphasising findings in proportion to their strength, and using cautious language where evidence is weak are all acts of honesty. A data story well told does not manipulate readers into a conclusion. It guides them through evidence, openly, so they can reach a sound conclusion for themselves.

Check: How does data storytelling guide readers, and why must it stay honest about limitations?

Section 46: Limitations of Analysis

Every analysis has limitations, and stating them plainly is a mark of quality, not a confession of failure. Limitations are the boundaries of what an analysis can support: the data it could not obtain, the period it could not cover, the methods it could not use, and the assumptions it had to make. No dataset is complete, no method is perfect, and no conclusion is free of uncertainty. A report that pretends otherwise asks readers to trust more than the evidence warrants.

Common limitations include coverage limits, when data covers only one region, one period, or one group. They include sample limits, when findings rest on a small or non-random sample. They include measurement limits, when a quantity was measured indirectly or inconsistently. They include method limits, when a method can describe association but cannot establish causation, or when a summary hides important variation. They also include missing data limits, when key fields were absent for many records and handling choices may have affected results. Naming the specific limitations that apply to your analysis is far more useful than a generic warning that results should be interpreted with caution.

Limitations must be stated plainly, in language readers can understand, and placed where they affect interpretation. A limitation buried in technical language at the end of a long report may never be read, and a limitation stated only after a strong conclusion has already been drawn may come too late to shape how that conclusion is received. The better practice is to mention a limitation when the affected finding is presented, and then to summarise all limitations together at the end. That repetition is not redundant. It ensures that limitations travel with the findings they qualify.

Limitations also guide next steps, which is why they are productive rather than merely defensive. A limitation about coverage suggests collecting data from the missing region or period. A limitation about sample size suggests a larger or more representative sample. A limitation about causation suggests a controlled comparison or a different method. A limitation about measurement suggests improving how a quantity is recorded. Each limitation points to a concrete improvement, turning a boundary of current knowledge into a plan for extending it.

Stating limitations protects decision makers as well. A decision based on a finding whose limits were hidden may be bolder than the evidence supports, with costs that fall on others. A decision based on a finding whose limits were stated can be calibrated: act now on the strong parts, wait or gather more data on the weak parts, and monitor outcomes where uncertainty remains. That calibration is the practical value of honesty about limits.

Finally, limitations protect the analyst. An analyst who states limits clearly cannot fairly be accused of overstating findings, because the report itself shows what was claimed and what was not. An analyst who hides limits may enjoy a brief appearance of certainty, but risks credibility when the limits are discovered later, as they usually are. Plain limitations, stated early and repeated where they matter, are the foundation of a reputation for trustworthy analysis, and that reputation is worth more than any single impressive result.

Check: Why must limitations be stated plainly, and how do they guide next steps?

Section 47: From Analysis to Machine Learning in Concept

Basic data analysis prepares for machine learning in several important ways, even though this chapter treats machine learning only conceptually. Machine learning, in broad terms, means building a program that learns patterns from data and uses those patterns to make predictions or decisions about new data. That learning depends entirely on the quality, structure, and understanding of the data it is trained on. Analysis is the stage where that understanding is built, and skipping it leads to models that learn the wrong patterns confidently.

The first preparation is data understanding. Analysis teaches you to ask what a dataset contains, where it came from, what period and population it covers, and what is missing. Those questions matter even more for machine learning, because a model trained on biased, incomplete, or mislabelled data will reproduce those flaws in its predictions. A model cannot know that its training data under-represented a group unless an analyst checks for that bias first. Analysis is therefore a safeguard, not a preliminary chore, before any learning begins.

Cleaning and preparation form the second link. Machine learning requires data in a consistent, numeric-friendly form, with missing values handled, categories represented consistently, and outliers investigated. Those are exactly the cleaning tasks covered in this chapter. A model trained on uncleaned data, with the same category spelled several ways or numbers stored as text, will treat fragments as distinct realities and produce unreliable results. The cleaning discipline of analysis transfers directly, because the data problems are the same, even when the eventual use is different.

Descriptive statistics and exploration form a third link. Summaries, distributions, and grouped comparisons reveal which fields vary, which fields relate to each other, and which patterns might be worth modelling. Exploration also reveals whether a prediction task is plausible at all. If no field relates to the outcome of interest in the observed data, a model is unlikely to find a reliable relationship by magic. Analysis sets realistic expectations for what learning can achieve, and it identifies the fields most likely to matter.

This chapter stays conceptual about machine learning for a deliberate reason. Machine learning involves its own methods, pitfalls, and evaluation practices, which require separate study. A conceptual overview is enough here to show the connection: clean, well-understood, honestly reported data is the foundation on which any learning rests. Claims that a model is accurate mean little without knowing what data it learned from, how that data was cleaned, and how its predictions were tested. Those questions are analysis questions, and they remain relevant whatever learning method is used.

The habit to carry forward is simple. Before building any model, perform the analysis steps of this chapter: ask a clear question, understand the dataset, clean it with documentation, explore its distributions and relationships, check for bias and limitations, and report what you found. A model built on that foundation starts from understanding rather than from hope, and its predictions can be judged with the same honesty that good analysis demands. Machine learning extends analysis. It does not replace it.

Check: How does basic analysis prepare for machine learning, and why is this chapter only conceptual about it?

Section 48: Analysis Practice Habits

Good analysis depends on habits, not only on techniques. A technique can be learned in an afternoon, but a habit shapes every analysis you perform, including the ones done under time pressure, when shortcuts are tempting. This section collects the habits that run through this chapter, so you can practise them deliberately until they become automatic.

The first habit is to start with a written question. Write the question down, decide what would count as an answer, and keep both beside your work. Return to them at the end and judge your findings against them. The second habit is to understand data before analysing it. Record its source, period, coverage, fields, and kinds, and note who or what is missing. The third habit is to verify loading and cleaning at every stage: row counts, column names, sample values, and a cleaning log that records what changed and why.

Further habits concern summaries and reporting. Always report spread with averages, group sizes with grouped summaries, and counts with percentages. Chart distributions before choosing typical values. Label every chart with a title, axes, and units. Check outliers, duplicates, and missing values before trusting a result, and document how each was handled. Separate evidence from interpretation, state uncertainty honestly, and name limitations where they affect conclusions. Those habits are individually small, but together they prevent most of the common pitfalls in analysis.

Reproducibility is a habit of its own. Keep original data unchanged, record steps in order, and test reproducibility by re-running from scratch before sharing. A notebook should be run top to bottom before it is shared. A script should be tested on known data before it is trusted on new data. Automation should include checks that flag unexpected changes, and its output should be reviewed by a person before it informs decisions. Reproducibility is not extra work added at the end. It is the way the work is done throughout.

Ethical habits belong in the same list. Collect only data you need. Protect privacy by aggregating and de-identifying where possible. Check for bias, especially when findings will affect people, and report suspected bias with its likely effect. Share findings only as widely as necessary when they are sensitive, and avoid selective reporting that presents only favourable results. Those habits protect the people behind the data, and they protect your credibility as an analyst.

Reviewing habits regularly is the final habit, and it closes the loop. After an analysis, ask which habits you followed well and which you skipped under pressure. After a mistake, ask which habit would have caught it, and strengthen that habit for next time. Habits decay when they are not reviewed, especially as tools change and new kinds of data appear. A short review after each project, noting one habit to keep and one to improve, is enough to keep the whole set alive. Analysis skill grows not from learning more techniques, but from practising reliable habits until they hold even when a deadline presses.

Check: What habits should every analysis follow, and why review those habits regularly?

Section 49: Review of Data Analysis Basics

Reviewing this chapter means returning to its central ideas and checking that you can apply them, not merely recognise them. Data analysis begins with a specific question, written down before data is examined, with a clear standard for what would count as an answer. It proceeds through understanding the dataset, verifying loading, cleaning with documentation, and handling missing values, duplicates, and outliers with investigation rather than automatic rules. Those early stages determine whether later summaries can be trusted, and they deserve as much attention as the summaries themselves.

The middle of the chapter concerns summaries and comparisons. Descriptive statistics describe a dataset in a few numbers, but they must be reported with context, including record counts and range. Mean, median, and mode describe typical values differently, and the choice among them should be justified by the distribution and the question. Spread must accompany every average, because datasets with the same average can differ greatly in variability. Distributions reveal shape, skew, and multiple peaks, and their shape affects which summaries are appropriate. Grouping, filtering, sorting, aggregation, joining, and pivot tables turn raw records into comparisons, provided group sizes are reported and row counts are verified at each step.

Charts and relationships form the next part of the review. Charts serve questions, and they must be labelled with titles, axes, and units. Chart types must match data types and questions, and distortions such as truncated bar axes and cherry-picked periods must be avoided. Trends and seasonal patterns must be distinguished from short-term fluctuation, and time series must be checked for gaps and measurement changes. Correlation describes association, not causation, and reporting language must reflect that limit, with outliers, groups, and third variables checked before a relationship is trusted.

The later sections concern trust and responsibility. Sampling must be random where possible, documented always, and large enough for its conclusions. Bias must be recognised, reported with its likely effect, and limited in its conclusions where it cannot be removed. Data quality, across accuracy, completeness, consistency, timeliness, and validity, must be reported honestly. Reproducibility must be tested by re-running from scratch. Notebooks must be run top to bottom before sharing. Libraries should be understood conceptually first, with current details learned from documentation. Ethics and privacy must shape collection, handling, and sharing of data about people.

Revisit pitfalls in particular, because pitfalls are where good intentions most often fail. Starting without a question, trusting unchecked data, reporting averages without spread, confusing correlation with causation, removing outliers automatically, and hiding limitations are errors that recur across projects and careers. A checklist that guards against them, used on every analysis, is a practical defence. Reviewing this chapter is not a matter of rereading it once. It is a matter of carrying its checklist into your next analysis, noticing where you are tempted to skip a step, and choosing to follow the step anyway. That choice, repeated, is what turns chapter knowledge into reliable practice.

Check: What should you review at the end of this chapter, and why revisit pitfalls in particular?

Section 50: Analysis Capstone Preview

This chapter has introduced the concepts of data analysis one by one, from asking a question to reporting findings honestly. The capstone preview looks ahead to how those concepts combine in a complete analysis that you will perform next. A complete analysis is not a single technique. It is a workflow in which every concept has a role, and in which the quality of the whole depends on the care taken at each stage.

The capstone begins, as every analysis should, with a question you write down before examining data. You will invent a simple practice dataset, such as daily sales of a few invented products over a month, and you will state one specific question about it. That question will guide every later choice, from which fields to include, to which groupings to try, to which charts to produce. Writing the question first is not a formality for the capstone. It is the step that makes the rest of the work analysable, because without it there is no standard against which findings can be judged.

Next, you will load and check the data, clean any deliberate inconsistencies you plant in it, and document what you changed. You will count missing values and decide how to handle them, check for duplicates, and investigate any outliers, recording each decision. Those steps apply the early chapters of this book in a practical setting, where the data is small enough to inspect by hand but the discipline is the same as for a large dataset. A capstone that skips cleaning would teach the wrong lesson, because it would suggest that summaries can be trusted without it.

Then you will compute descriptive summaries, including a typical value and its spread, and you will group the data by one categorical field, reporting group sizes alongside the summaries. You will produce at least two honestly labelled charts of different types, choosing chart types that match your question and your data, with titles, axes, and units. You will check whether patterns differ across groups, whether a trend over time is sustained or merely a short fluctuation, and whether any apparent relationship is correlation rather than causation. Each of those checks draws on a different section of this chapter, and together they show why the concepts were taught separately: so they can be combined thoughtfully.

Finally, you will write a short report that states your question, your data and its limitations, your findings with charts, and what you would investigate next. The report will separate evidence from interpretation, state uncertainty honestly, and avoid claiming causation from correlation alone. It will also respect privacy, because your data is invented practice data with no real personal information. That constraint is part of the exercise, because it reminds you that analysis can be practised fully without using real people's data.

The capstone is a preview, not a test of perfection. Its value lies in performing the whole workflow once, end to end, and noticing where the steps connect. Cleaning affects summaries. Grouping affects interpretation. Chart choices affect what readers see. Limitations affect what conclusions are safe. When you have performed that complete cycle, the concepts of this chapter will no longer be separate topics. They will be a single practice you can repeat, refine, and trust.

Check: How will the analysis concepts in this chapter combine in a complete analysis you perform next?

Practical Task

Perform a small complete analysis on a dataset you create yourself, using only invented practice data with no real personal information: invent a simple table of at least 30 rows, such as daily sales of a few invented products over a month, write down one specific question before examining it, load and check the data, clean any deliberate inconsistencies you plant in it, compute descriptive summaries including a typical value and its spread, group the data by one categorical field with group sizes reported, produce at least two honestly labelled charts of different types, check for outliers and missing values and document how you handled them, and write a short report that states your question, your data and its limitations, your findings with charts, and what you would investigate next, without claiming causation from correlation alone.

Chapter Questions

1. Why must you write a specific analysis question and decide what counts as an answer before examining data?
2. How do mean, median and mode differ, and why must spread be reported alongside any typical value?
3. Explain grouping and aggregation: what do they reveal that overall summaries hide, and what checks must accompany them?
4. Why does correlation not prove causation, and how should that caution shape both reporting language and decisions?
5. What makes an analysis reproducible and honest, including its handling of missing values, bias, limitations and privacy?