How a web page loads and renders

From URL to bytes#

A URL has parts, and the browser uses them in order:

https://shop.example.com:443/search?q=rust#results scheme   host            port path    query  fragment

From HTML to something you can click#

The steps overlap: the browser starts parsing before the download ends.

1. The DOM. The parser reads the HTML as it streams in and builds the Document Object Model: a tree with one node per element, text and attribute. This tree, not the HTML text, is what scripts and automation tools work with. It is also what a selector such as #go or h1 searches.
2. Subresources. As the parser meets <link rel="stylesheet">, <script src>, <img> and fonts, it starts downloading each one in parallel. A typical page makes tens to hundreds of requests, many to other hosts, each going through the steps above.
3. CSS. Stylesheets become a style tree; together with the DOM they decide each element’s size, position, colour and whether it is shown at all.
4. Layout and paint. The browser computes where every box goes (layout), then draws the pixels (paint) and composes the layers on screen. An element can exist in the DOM and still be invisible, zero-sized, or behind another.
5. JavaScript. A classic <script> blocks the parser until it has downloaded and run, because it might change the document. Scripts marked defer run after parsing; async ones run as soon as they arrive. Scripts can fetch more data (fetch, XMLHttpRequest), insert elements, and react to clicks.

Modern sites are often single-page applications: the first HTML is nearly empty, and JavaScript downloads data and draws the whole interface. Moving to another “page” inside the app changes the URL without loading a new document at all.

From bytes to pixels#

Once the bytes of a page have arrived, the browser still has to turn them into something you can see. It does that in five steps, called the rendering pipeline.

The rendering pipeline. Later steps depend on earlier ones.

Reflow and repaint#

After the first frame, the page changes: a script edits the DOM, the user hovers a button, an image finishes loading. The browser then redoes only the steps the change needs. Cheaper changes skip steps.

A reflow (layout again) can spread: changing one element near the top may move everything below it. A repaint redraws pixels without moving anything. Two habits make pages slow:

• Layout thrashing. A script writes to the DOM, then reads a layout value such as offsetHeight, then writes again, in a loop. Each read forces the browser to run layout at once, so the work is done many times instead of once. Reading everything first and writing afterwards avoids it.
• Animating the wrong properties. Animating left or height redoes layout on every frame. Animating transform and opacity only composites.

The events, and what “loaded” means#

The browser reports progress with a few events and a readyState:

“The page loaded” is therefore a precise but small claim: the document and the files it names have arrived. It says nothing about the data that scripts fetch afterwards, the spinner that is still turning, or the button that only works once its script has attached a handler.

Automation tools expose the same moments under their own names:

Why automation needs waits#

Put the pieces together and the problem shows. When goto returns, the page has “loaded”, yet:

• the list of results may still be a request in flight;
• the element you want may not exist yet, or exists but is hidden;
• a click handler may not be attached, so a click does nothing;
• an overlay (a cookie banner, a spinner) may cover the target;
• the framework may replace the element you just found (a stale reference).

None of these is an error. They are the normal state of a page between “loaded” and “ready”. A test that reads the page the moment load fires will sometimes win this race and sometimes lose it, which is why it is flaky. The fix is to wait for a condition the test actually needs:

• an element to exist, to be visible, or to be clickable;
• a text to appear, or a URL to change;
• the network to go quiet, when there is nothing specific to wait for.

SeleniumBase and the pure CDP engine build this in: their action methods and Locators wait for the element before they act, up to a timeout, and Locators look the element up again each time, so the stale-reference problem does not arise. See WebDriver waits and the CDP engine.

Animations#

Motion on a page comes from four sources, and a test can meet all of them.

Why animations make tests flaky#

A test acts at machine speed, so it often acts in the middle of a movement.

• A moving target. A menu slides in over 300 ms. The test finds the button, measures its position, and clicks there, but by then the button has moved. The click lands on whatever is under that spot, or the browser reports that the element is covered.
• A half-visible element. A fade-in reaches opacity: 0.4 when the test checks for visibility. Whether that counts as visible depends on the tool.
• Different speed on different machines. On a slow CI runner a 300 ms animation can take twice as long, so a fixed sleep that works on a laptop fails there.
• Looping animations. A spinner never ends, so “wait until nothing changes” never succeeds.
• Screenshots. A visual comparison catches a frame from the middle of the motion, and the images differ every run.

The usual fixes, from best to worst:

1. Wait for the state the animation leads to (the menu is open, the spinner is gone), not for time.
2. Ask the page for less motion. Many sites honour the CSS media query prefers-reduced-motion: reduce, which the user sets in the operating system.
3. Switch the animations off for the test run with a style rule:

// Add a stylesheet that removes motion, then finish what is already running.const style = document.createElement('style');style.textContent = '*, *::before, *::after { animation: none !important; transition: none !important; }';document.head.appendChild(style);document.getAnimations().forEach(animation => animation.finish());

Other things that can happen#

• iframes embed another document inside the page. Each has its own DOM, and often a different origin, so automation must switch into it (WebDriver) or attach to it as a separate target (CDP).
• Shadow DOM hides part of a component’s tree from ordinary selectors. See Shadow DOM.
• Popups and new tabs are new windows, with their own load.
• Service workers can answer requests from a cache without the network at all, so a “load” may never touch the server.
• Cookies and storage change as the page loads (Set-Cookie, localStorage writes); see Sessions, cookies and storage.

Core Web Vitals#

Core Web Vitals are three numbers Google chose to describe how a page feels. They are measured in real browsers and are one signal used by search (see SEO).

Typical causes:

• Slow LCP: a slow server (see network), a big unoptimised image, a stylesheet or script that blocks rendering, content that only appears after JavaScript fetches it (see frameworks and the DOM).
• Slow INP: long tasks on the main thread. JavaScript, style, layout and paint share one thread, so a 400 ms script means the click waits 400 ms.
• High CLS: images or ads without a reserved size, fonts that swap late, banners inserted above content that has already been read.

Two kinds of data exist. Field data comes from real visitors and is what Google ranks on. Lab data comes from one controlled run, as in a test or in Lighthouse. Lab data is repeatable, but it is a model: it has no real INP, since nobody clicks.

The Performance API#

Every browser records timings and exposes them to JavaScript through the performance object. You can read the same numbers from a test.

// How the page load went, in milliseconds since the navigation started.const nav = performance.getEntriesByType('navigation')[0];({ ttfb: nav.responseStart, domReady: nav.domContentLoadedEventEnd, load: nav.loadEventEnd });// When the first paint and first contentful paint happened.performance.getEntriesByType('paint');// Every request the page made, with its size and duration.performance.getEntriesByType('resource').map(r => [r.name, Math.round(r.duration)]);

Some metrics are announced as they happen rather than stored. A PerformanceObserver subscribes to them: largest-contentful-paint, layout-shift (the pieces of CLS), longtask, event. Observers are described in Web APIs.

Lighthouse#

Lighthouse is Google’s audit tool, built into Chrome DevTools and also available on the command line. It loads a page in a controlled, throttled setting, collects metrics and gives a 0 to 100 score for performance, accessibility (see accessibility), best practices and SEO. It drives Chrome through the DevTools Protocol. Treat the score as a trend, not a verdict: it moves a few points between runs, and the list of recommendations is more useful than the number.

Making a slow machine behave like a slow phone#

To see how a page behaves on a weak device, ask Chrome to slow itself down with two DevTools Protocol commands: Emulation.setCPUThrottlingRate (a rate of 4 makes the CPU four times slower) and Network.emulateNetworkConditions (latency and bandwidth; see network). A page that is fine on a fast laptop often shows its long tasks and layout shifts under throttling.

How seleniumbase-rs uses it#

seleniumbase-rs does not ship a Web Vitals or Lighthouse integration. What it gives you are the tools to build one in a few lines.

• Waiting for the load to finish. BaseCase::wait_for_ready_state_complete polls document.readyState until it is complete, within the default wait, and fails with a wait timeout otherwise. It waits for the load event, not for the page to be visually stable.
• Reading the browser’s measurements. BaseCase::execute_script (WebDriver engine) and sb_cdp::Page::evaluate (pure CDP engine) run JavaScript and return the JSON-representable result. Page::evaluate awaits promises, so an observer that resolves a promise works.
• Throttling. BaseCase::execute_cdp_with_params sends any DevTools command by name.

use seleniumbase_rs::sb_cdp::{Browser, LaunchOptions};let browser = Browser::launch(LaunchOptions::builder().headless(true).build()?).await?;let page = browser.default_page().await?;page.goto("https://example.com").await?;// LCP: the observer replays what already happened (buffered), and the promise is awaited.let lcp = page    .evaluate(        r#"new Promise(resolve => {             new PerformanceObserver(list => {               const entries = list.getEntries();               resolve(entries[entries.length - 1].startTime);             }).observe({ type: 'largest-contentful-paint', buffered: true });           })"#,    )    .await?;assert!(lcp.as_f64().unwrap() < 2500.0, "LCP too slow: {lcp}");browser.close().await?;

use serde_json::json;sb.open("https://example.com").await?;sb.wait_for_ready_state_complete().await?;// Slow the CPU down four times for the rest of the session.sb.execute_cdp_with_params("Emulation.setCPUThrottlingRate", json!({ "rate": 4 })).await?;let nav = sb    .execute_script("return performance.getEntriesByType('navigation')[0].responseStart;")    .await?;println!("time to first byte: {nav} ms");

Where to go next#