Crawlers, spiders and SEO

Crawling, scraping and testing#

The three words are often mixed up. They differ in what the program is for.

Many tools do two of them. A link checker crawls to find pages and tests each one. A scraper often crawls to find the pages it will read.

How a crawler works#

A crawler is a loop around a queue, the crawl frontier, which holds the URLs that are known but not yet visited.

The crawl loop. It stops when the frontier is empty or a limit is reached.

The order of the frontier#

A queue (first in, first out) visits pages breadth first: everything one click from the seed, then two clicks away. A stack goes depth first, deep into one branch. A priority queue visits the pages that matter most first, such as those with many incoming links, and is what large crawlers use.

Normalization and traps#

Two addresses can show the same page, and a crawler has to see through that: #section fragments are removed, the host name is lower-cased, tracking parameters such as utm_source are dropped, and parameters are put in a fixed order. A page can also declare its preferred address with link rel="canonical". Without this work a crawler finds a trap: a calendar with a “next month” link forever, a faceted search with millions of combinations, or a session id in every URL. A depth limit, a page limit and a rule on the number of parameters keep a crawl finite.

What counts as a link#

Mostly a href. But not every href is a page: mailto:, tel: and javascript: are not crawled, and # points to the same page. Links can also sit in area, in forms that use GET, in iframe src, and, in modern apps, only in JavaScript click handlers that never appear in the HTML. A link marked rel="nofollow" asks crawlers not to count it.

robots.txt and sitemaps#

A site tells well-behaved crawlers what it wants in two small files at known addresses.

robots.txt lives at /robots.txt and follows the Robots Exclusion Protocol. A crawler fetches it first, and checks every URL against it before requesting the page:

User-agent: *Disallow: /admin/Disallow: /searchAllow: /search/helpUser-agent: BadBotDisallow: /Sitemap: https://example.com/sitemap.xml

Rules are grouped by User-agent, the most specific path wins, and Sitemap points to the next file. Some crawlers also read a non-standard Crawl-delay. Two points matter: robots.txt is a request, not a lock, because nothing stops a program that ignores it, and a page you do not want in a search index needs a noindex meta tag or an X-Robots-Tag header, since a disallowed URL can still be listed if others link to it.

A sitemap is an XML list of the URLs a site wants found, with optional last-change dates. One file may hold up to 50,000 URLs, and a sitemap index lists several files.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">  <url>    <loc>https://example.com/products/1</loc>    <lastmod>2026-09-30</lastmod>  </url></urlset>

For a test of your own site, the sitemap is the best seed list, because it names the pages that are meant to exist.

Politeness#

A crawler uses someone else’s server, so it follows the rules of a good guest.

• Identify yourself. Set a User-Agent that names the program and gives a contact, so an operator can reach you instead of blocking you.
• Limit the rate per host. One request a second or slower is a common default, with a small, fixed number of simultaneous connections to one host. Fast crawling is done by visiting many hosts, not by pressing one hard.
• Back off. On a 429, a 503 or slow answers, wait longer, and double the wait after each failure.
• Do not fetch twice. Keep the ETag and Last-Modified of a page and ask again with If-None-Match or If-Modified-Since; the server can answer 304 with no body.
• Respect robots.txt, and run big crawls when the site is quiet.

Pages that need JavaScript#

A plain HTTP client gets the HTML the server sends. On a modern single-page app that HTML is almost empty, and the links and content appear only after the scripts run. For those pages the crawler needs a browser.

A common design is hybrid: fetch with the cheap client, notice that the page is a shell with no useful text, and render only those pages. With a browser, the crawler must also decide when the page is ready (see how a page loads), scroll for lazy-loaded content, and close the page afterwards so memory does not grow.

Responsible use#

Crawl, index, rank#

The path of a page into search results.

What a page must say#

Almost everything below is in the head of the HTML.

Also part of the page: readable URLs (/shoes/trail-runner, not /p?id=88213), links with descriptive text, not “click here”, and content that is useful to read. Headings, alt text and a good title also help assistive technology (see accessibility).

Canonical URLs#

The same page often has many addresses: with and without www, http and https, a trailing slash, tracking parameters (?utm_source=...), a sort order. Search engines would treat them as separate pages and split the credit. The canonical link names the main one:

<link rel="canonical" href="https://example.com/shoes/trail-runner">

It is a strong hint, not an order. A permanent redirect (status 301) is the stronger tool when an address should simply disappear.

Structured data (JSON-LD)#

Structured data states facts in a form a machine can read, using the vocabulary of schema.org. The usual format is JSON-LD, a script block that holds JSON and is not displayed:

<script type="application/ld+json">{  "@context": "https://schema.org",  "@type": "Product",  "name": "Trail Runner",  "offers": { "@type": "Offer", "price": "89.00", "priceCurrency": "EUR" }}</script>

It can earn rich results (stars, prices, recipe times, FAQ entries) in the listing. Common types are Product, Article, Recipe, Event, BreadcrumbList and Organization. The data must match what the visitor can see; invalid or misleading markup is ignored or penalised. Google’s Rich Results Test and the Schema Markup Validator check it.

Robots, sitemaps and status codes#

Three tools look alike and do different jobs.

The HTTP status code is read too: 200 is a page, 301 a permanent move, 302 a temporary one, 404 and 410 mean gone, 5xx means the server failed and the bot should come back later. A page that says “not found” but answers 200 (a soft 404) confuses engines. Links can carry rel="nofollow", rel="sponsored" or rel="ugc" to withhold credit, and hreflang points to the translations of a page.

Server rendering or client rendering#

Compare what a bot receives with what a browser shows (see frameworks and the DOM and modern web).

Search engines are better at JavaScript than they were, but other consumers are not: the preview card for a shared link, some smaller search engines and many AI crawlers read the first HTML only. The simplest check is to compare View Source (the response as sent) with the Elements panel (the DOM after the scripts). If the title, the headings and the text are missing in the first, you depend on rendering.

Core Web Vitals#

Speed and stability are a ranking signal, grouped as page experience and measured by the Core Web Vitals (LCP, INP and CLS; see rendering and performance). They matter most as a tie-breaker between pages of similar relevance: a fast page with poor content will not beat a slow page that answers the question.

Checking SEO in automated tests#

SEO bugs are quiet: nothing crashes, the traffic just goes. The most expensive ones are easy to test.

• Production carries noindex. A staging setting leaks into the release and the whole site drops out of search. Assert that the robots tag does not contain noindex on production, and that it does on staging.
• The canonical is wrong: it points to staging, to a relative path, or to the home page for every page.
• The title or the description is missing, empty or duplicated by a template bug.
• No single h1, or the text appears only after a script runs.
• JSON-LD is broken and no longer parses.
• robots.txt blocks everything, left over from development.
• A redirect chain or a 404 on a page that is in the sitemap.

Check the rendered page (what the browser shows) and, for the content that must not depend on scripts, the raw HTML as an ordinary HTTP client such as curl gets it.

How seleniumbase-rs uses it#

Link checks and crawling#

The library does not have a crawler today: no frontier, no depth limit, no robots.txt parser. What it has are the building blocks of a link check, on BaseCase:

• get_all_links() returns the href of every a element on the current page.
• get_link_status_code(url) returns the HTTP status of a link. It sends a HEAD request, repeats it as GET when the server answers 405 or 501, and does not follow redirects, so a moved page shows as 301 or 302. This matches Python SeleniumBase.
• assert_link_status_code_is_not_404(url) fails when the answer is 404.
• print_unique_links_with_status_codes() prints link -> code for each unique link on the page, checking eight at a time, and lists non-HTTP links such as mailto: as skipped.

sb.open("https://example.com").await?;// Check one page: every same-site link must not be a 404.let links = sb.get_all_links().await?;for link in links.iter().filter(|l| l.starts_with("https://example.com")) {    sb.assert_link_status_code_is_not_404(link).await?;}// Or just print a report.sb.print_unique_links_with_status_codes().await?;

To check a whole site, you write the outer loop yourself: read the sitemap, open each page, and run the check, keeping the politeness rules above.

Pages that need JavaScript work with the ordinary engines, because the link list comes from the live DOM after the scripts have run.

SEO checks in tests#

The library has no SEO audit, and nothing to check a canonical link, a meta description, the robots tag or JSON-LD directly. It gives you the pieces to read them from the live page.

• Title. BaseCase::get_title, assert_title(expected) and assert_title_contains(expected). The pure CDP engine has Page::title() and page expectations to_have_title and to_contain_title.
• Tags in the head. BaseCase::get_attribute(css, attribute) returns the attribute of an element as an Option<String>. It first waits for the element to be present, so it suits tags that must exist. For a tag that may be absent, read it with execute_script, or Page::evaluate on the pure CDP engine.
• Basic structure. HtmlInspector::inspect(html) has rules that double as SEO checks: document-title (no title, or an empty one), document-lang, meta-viewport, page-h1 (no h1, or more than one), image-alt, heading-order and link-text. The MCP tool audit_page runs the same rules on the open page (see accessibility and MCP).

use seleniumbase_rs::api::html_inspector::HtmlInspector;sb.open("https://example.com/shoes/trail-runner").await?;sb.assert_title_contains("Trail Runner").await?;// Tags that must exist: get_attribute waits for them.let description = sb.get_attribute("meta[name=description]", "content").await?;assert!(description.is_some_and(|text| (50..=160).contains(&text.len())));let canonical = sb.get_attribute("link[rel=canonical]", "href").await?;assert_eq!(canonical.as_deref(), Some("https://example.com/shoes/trail-runner"));// A tag that may be absent: read it with a script instead.let robots = sb    .execute_script("return document.querySelector('meta[name=robots]')?.content ?? '';")    .await?;assert!(!robots.as_str().unwrap_or_default().contains("noindex"));// The structural rules, on the DOM as it is now.let html = sb.execute_script("return document.documentElement.outerHTML;").await?;let result = HtmlInspector::inspect(html.as_str().unwrap_or_default())?;let seo_rules = ["document-title", "page-h1", "document-lang"];assert!(result.issues.iter().all(|issue| !seo_rules.contains(&issue.rule.as_str())));

use serde_json::Value;page.goto("https://example.com/shoes/trail-runner").await?;assert!(page.title().await?.contains("Trail Runner"));// Every JSON-LD block must parse and have a type.let blocks: Vec<Value> = page    .evaluate_as(        "[...document.querySelectorAll('script[type=\"application/ld+json\"]')]           .map(script => JSON.parse(script.textContent))",    )    .await?;assert!(!blocks.is_empty() && blocks.iter().all(|block| block["@type"].is_string()));

Where to go next#