C++ Web Crawler Project
Start from one page, follow every link, and map a slice of the web.
What you will build
Give the crawler a starting URL and a depth; it fetches pages, extracts links, and visits each new one exactly once — breadth-first, like a search engine's scout. You'll wrestle with real-world mess: redirects, relative URLs, robots etiquette, and pages that never answer.
What you will learn
- HTTP requests (libcurl or raw sockets)
- Extracting href links from HTML
- Breadth-first traversal with a queue
- Avoiding revisits with a visited set
- Respecting robots.txt basics and rate limits
Feature checklist
- Start URL and max-depth setting
- Fetches pages and extracts all links
- Visits each URL once (normalisation included)
- Same-domain-only mode
- Prints a crawl tree and page count
- Polite delay between requests
Starter code
#include <iostream>
#include <queue>
#include <string>
#include <unordered_set>
// fetch(url) -> page HTML (implement with libcurl).
std::string fetch(const std::string& url);
std::vector<std::string> extractLinks(const std::string& html,
const std::string& base);
int main() {
std::queue<std::pair<std::string, int>> q; // url, depth
std::unordered_set<std::string> seen;
const int MAX_DEPTH = 2;
q.push({"https://example.com", 0});
seen.insert("https://example.com");
int pages = 0;
while (!q.empty()) {
auto [url, depth] = q.front(); q.pop();
std::string html = fetch(url);
if (html.empty()) continue;
++pages;
std::cout << std::string(depth * 2, ' ') << url << "\n";
if (depth == MAX_DEPTH) continue;
for (auto& link : extractLinks(html, url))
if (seen.insert(link).second) // true only if new
q.push({link, depth + 1});
}
std::cout << "Crawled " << pages << " pages.\n";
}Full walkthrough
Step 1: Split the problem at the fetch() seam
The starter declares std::string fetch(url) (implemented with libcurl: curl_easy_init, CURLOPT_URL, a write callback appending into a std::string, curl_easy_perform) and std::vector<std::string> extractLinks(html, base). The crawler logic never touches curl — that seam means you can test extractLinks on saved HTML without any network at all.
Step 2: Represent the frontier as a queue of (url, depth)
Breadth-first crawling uses std::queue<std::pair<std::string, int>>: pop a URL, fetch it, extract its links, push each unseen link at depth + 1. BFS (queue) rather than DFS (stack) is what makes 'max-depth' mean 'distance from the start page', which is the promise the user set. MAX_DEPTH starts at 2.
Step 3: Never visit twice: normalise, then check the set
An std::unordered_set<std::string> seen guards every push. But raw URLs lie: 'http://x.com/a', 'http://x.com/a#top', and 'HTTP://X.COM/a' are one page. Normalise before inserting — lowercase the scheme and host, strip the fragment, resolve relative links against the page's base URL. Skipping normalisation is why beginner crawlers loop forever on the same page wearing different hats.
if (seen.insert(normalised).second)
q.push({normalised, depth + 1}); // only queue genuinely new pagesStep 4: extractLinks(): honest string surgery
Scan the HTML for 'href="' occurrences, take characters until the closing quote, keep the ones starting with http (or resolve relative ones against base), and return them. It is not a real HTML parser and will miss exotic markup — say so in a comment. Knowing WHERE your tool is approximate is engineering honesty, and for link discovery this approximation catches the overwhelming majority of real links.
Step 5: Same-domain-only mode
Extract the start URL's host once; before queueing a link, compare its host. One string comparison turns a crawler that wanders the entire web into a site-mapping tool — the difference between a useful audit of YOUR site and an accidental DDoS of someone else's.
Step 6: Be polite: delay between requests
Sleep ~500 ms (std::this_thread::sleep_for) between fetches. A crawler is a guest hammering someone's server; the delay is both courtesy and self-preservation (servers ban rude crawlers by IP). Print progress as you go — URL and depth — so a long crawl is observable, not a hang.
Step 7: Print the crawl tree and counts
Indent each printed URL by its depth so the output is a tree, then summarise: pages visited, links skipped as duplicates, links skipped as off-domain, fetches that failed. The skipped counts are the interesting half of the report — they prove the seen-set and the domain filter actually fired.
Common mistakes
- Queueing links without a seen-set, so two pages linking to each other keep the crawler bouncing forever.
- Pushing the raw href text — relative links like '/about' are not fetchable URLs until resolved against the page's base.
- Treating one failed fetch as fatal; real crawls lose pages constantly and must log, skip, and continue.
Compile and run
g++ -std=c++17 -o crawler crawler.cpp -lcurlNeeds libcurl installed (Linux: sudo apt install libcurl4-openssl-dev; macOS: ships with the Xcode tools / brew install curl). The -lcurl flag links it. Crawling other people's sites has rules: keep the polite delay, respect robots.txt on real crawls, and only crawl sites you have a right to test.
Sample run
$ ./crawler
Start URL:
> https://example.com
Max depth:
> 2
Same domain only? (y/n):
> y
Crawling (500 ms polite delay between pages)...
https://example.com/
https://example.com/about
https://example.com/docs
https://example.com/docs/install
https://example.com/docs/usage
https://example.com/about (already seen — skipped)
https://iana.org/domains (different domain — skipped)
Done. Visited 5 pages · skipped 1 duplicate · skipped 1 off-domain · 0 failed.Try it in your browser
No browser demo is included for this project. Build the C++ version below — that is the real program.