Benchmarking - SeleniumBase for Rust

The repository has two benchmarks in benches/. They measure what the library itself costs: its own CPU work, and what it adds on top of a Chrome that is already running. This page says how to run them and how to compare a change. The measured figures are on Benchmarks, and advice for making your own tests faster is on Performance.

Info

What you will learn

What cpu and cdp_latency measure, and what each needs.
How to run them, and where their results go.
How the published figures were produced.
How to compare before and after a change, and keep the runs fair.

The two benchmarks#

Both are plain programs, not a framework. Each [[bench]] in Cargo.toml sets harness = false, and each file has its own main that times with std::time::Instant and prints Markdown tables. There is no criterion or divan, and no saved baseline.

Bench	Measures	Needs	Command
`cpu`	The library’s own work	no browser	`cargo bench --bench cpu`
`cpu` with `turso`	The same, plus the embedded database	no browser	`cargo bench --bench cpu --features turso`
`cdp_latency`	The pure CDP engine against a real Chrome	Chrome or Chromium	`cargo bench --bench cdp_latency`

`cpu`#

It times, with no browser involved:

Fingerprint::randomized for a Windows persona, building the stealth bootstrap script, and Fingerprint::validate;
classifying a CSS or XPath selector (Selector::auto, eight selectors per call);
planning a human mouse path (600 px) and human typing (100 characters);
HtmlInspector::inspect on a page of 50 rows and one of 2,000 rows.

Each operation is warmed up (up to 50 calls), then timed in nine batches of a fixed number of calls. The figure printed is the median batch divided by the number of calls, with the calls per second next to it. std::hint::black_box keeps the compiler from removing the work.

With --features turso a second table follows, from an in-memory ResultStore and ProfileVault: recording a test result, the flaky-test query, opening a vault, and sealing and loading a profile. Each is one timed loop, not nine batches. Opening the vault is slow on purpose: that is the key derivation.

`cdp_latency`#

It launches Chrome with the pure CDP engine, headless, and serves a fixed page on a loopback port, so no outside network is involved. What it prints is the cost the library adds to a browser that is already running.

Table	What is timed	Runs
Start-up	Launching Chrome to a DevTools connection, then attaching and loading the first page	5 launches
One tab, one command at a time	`evaluate`, a locator count, reading text, a click, `fill`, typing key by key, a viewport screenshot, `goto` a local page; each warmed up 10 times first	30 to 500 per operation
How long a wait takes to notice an element	The element appears 100 ms after a script runs; the page’s own clock measures how late the wait sees it, including one round trip	40
Many tabs at once	1, 2, 4 and 8 tabs in one Chrome, each clicking a button 150 times at the same time: wall time and clicks per second	one pass per tab count
Memory of the driving process	Resident memory of the benchmark process, not Chrome’s, with 0, 1, 4 and 8 browsers	one reading each

These variables change how it runs:

CHROME_BINpath#

The Chrome to launch. CHROMIUM_BIN is tried next, then the usual install locations, as the pure CDP engine always does.

CIany value#

When set, Chrome is launched with --no-sandbox, which it needs as root or in a container.

RUSTC_VERSIONtext, at build time#

Read when the bench is compiled, for the header line: “Rust stable” when it is unset. Set it to label a run, for example RUSTC_VERSION="$(rustc --version)" cargo bench --bench cdp_latency.

Note

The memory table reads ps -o rss=. Where there is no ps, as on Windows, it shows 0.

Run them#

Quiet the machine

Close other builds, browsers and anything else that uses the CPU. On a laptop, use mains power. Timings from a loaded machine mean little.

Build and run

Terminal

cargo bench --bench cpucargo bench --bench cdp_latency

cargo bench builds with Cargo’s bench profile, which inherits release; neither manifest sets a profile of its own. Always name --bench: a bare cargo bench also builds the library and the binaries as benchmark targets, which takes longer and adds nothing, since neither has a benchmark.

Keep the output

The tables go to standard output, and cargo’s own messages to standard error. Nothing is written anywhere else, so redirect the output to keep it:

Terminal

cargo bench --bench cdp_latency > target/bench-cdp.md

target/ is ignored by git, so the file stays out of commits.

How the published figures were made#

Benchmarks shows one run of each harness on one machine. That page names the machine, the Chrome version, the date and the commit, and says what is uncertain about each figure. The runs were release builds, headless, against a local page, with nothing else running. The tables there are the harness tables with some columns left out, and Performance quotes a few of their figures.

Warning

Do not copy those figures into a new page or a pull request as if they were yours. Run the harnesses on your machine, and quote what you measured with the machine and the command.

Compare before and after a change#

Nothing in the repository compares two runs for you. Do it by hand, the same way each time.

Measure the baseline

Terminal

git switch maincargo bench --bench cpu > target/bench-before-1.md

Terminal

git stashcargo bench --bench cpu > target/bench-before-1.mdgit stash pop

Measure the change

Terminal

git switch my-branch    # not needed for an uncommitted changecargo bench --bench cpu > target/bench-after-1.md

Repeat and compare

Run each side at least three times, alternating before and after, and compare the medians row by row, for example with diff target/bench-before-1.md target/bench-after-1.md. A difference smaller than the spread between repeated runs of the same commit is noise.

Warning

To keep the baseline in a second worktree instead, give each worktree its own target directory. Two checkouts that share one CARGO_TARGET_DIR overwrite each other’s compiled library.

When a pull request is about speed, put the before and after figures in its description, with the machine, the Chrome version and the command.

Keep the runs fair#

An optimised build. Time cargo bench only, never a debug build.
The same everything. The same machine, power settings, Chrome version (for cdp_latency), features and command on both sides.
No background load. A single run is a sample: on a busy machine, the same table can move a lot between two runs.
Headless only. The harness always runs Chrome headless. Real-Chrome timings differ between headless and headed runs, so do not compare one with the other.
Cold starts vary. The start-up table launches Chrome five times; a cold start can take much longer than the median.
Containers. Inside one, set CI for --no-sandbox, and keep its memory limit the same for both sides.

In CI#

Note

No CI job runs the benchmarks. .github/workflows/ builds, tests, lints, checks the minimum Rust version, the dependency policy and the docs, but has no cargo bench. The Dockerfile only creates empty benches/cdp_latency.rs and benches/cpu.rs in its skeleton step.

Add a benchmark#

Write the program

Add a file to benches/ with its own main, as the two that exist: warm up, time several runs, and print the median as a Markdown table.

Declare it

Add a [[bench]] with harness = false to Cargo.toml.

List it in the Dockerfile

Add the file to the skeleton step of the Dockerfile, or the image build fails; tests/dockerfile_skeleton.rs checks it.

Next steps#

Benchmarks

The figures from these harnesses, and what changed because of them.

Performance

How to make your own tests faster, and how to measure them.

Build from source

The toolchain, the build settings and the checks.

Debugging

Logs, evidence on failure, and a browser you can watch.

The two benchmarks#

cpu#

cdp_latency#

Run them#

How the published figures were made#

Compare before and after a change#

Keep the runs fair#

In CI#

Add a benchmark#

Next steps#

`cpu`#

`cdp_latency`#