Stealth Scraping

A site can tell a client apart before any page script runs: by the TLS ClientHello, by the HTTP/2 connection start, and by the order and spelling of the request headers. Fingerprint & Stealth Profiles covers what a page can read from inside the browser. This page covers the other side, the wire, and the pieces that keep an identity, its cookies and its exit address together while a scraper works through a site’s defences.

Everything here is young, and it is not a stable API yet: types and paths may change between releases. The parts that exist are tested against local captures and a local Chrome, on one system. Read Limits before you rely on any of it.

Status#

Features#

What the stealth layer does today, in plain words. Each claim was checked against the code. All of it is experimental, measured only for Chrome 155 on macOS, and tested against local servers and a local Chrome, not against live services.

Measured fingerprints, checked by a round trip#

• The capture tools (capture_tls and capture_hints in scrape, capture_h2 in mimic) record the TLS ClientHello, the client hints, User-Agent, Accept-Language, the HTTP/2 connection start and the header lists of a local Chrome. They write crates/codec/src/measured/chrome-155-macos.json and keep the raw bytes as test fixtures.
• Nothing is typed by hand. seleniumbase_rs_codec::emulation embeds that file. A system that was not measured gets NotMeasured, and a Chrome of another major version gets VersionMismatch; no other system’s values are borrowed.
• The round trip is capture, emulate, compare. The wire test builds a client from the profile, connects it to a loopback TLS server of the crate, reads what arrived with the crate’s own parsers (ClientHello, h2_frames) and compares it with the measurement: the TLS spec equals the measured one minus two known gaps (KnownGap::TrustAnchorsExtension and KnownGap::MlDsaSignatureAlgorithms, which BoringSSL and wreq cannot send), the Akamai HTTP/2 string equals the measured one, and the headers arrive in the measured order and spelling. The committed HTTP/2 spec is also derived again from the stored evidence and the raw frame fixtures.

TLS, JA4 and HTTP/2 emulation through a MITM proxy#

MitmProxy::start binds a proxy on 127.0.0.1 for a real Chrome. It ends Chrome’s TLS with a leaf certificate of an in-memory CA (EphemeralCa), which Chrome trusts through --ignore-certificate-errors-spki-list with the leaf key pin (never --ignore-certificate-errors; the CA is not installed anywhere). It then replays each request through the impersonating wreq client, so the site sees the measured ClientHello (and so the measured JA4), the measured HTTP/2 SETTINGS and their order, the connection window, the PRIORITY frames, the HEADERS priority, the pseudo-header order, and the header order and spelling, plus Chrome’s priority header from the measured table. Bodies are streamed both ways within size limits. A wss:// or ws:// WebSocket handshake is relayed with the measured WebSocket ClientHello. By default the proxy refuses loopback, private and cloud-metadata destinations. Chrome’s own TLS handshake ends at the proxy, so the proxy is the one the site sees.

A plain HTTP client without a browser#

impersonation_client(&profile) takes an ImpersonationProfile and returns a wreq::ClientBuilder (or an ImpersonateError) with the measured TLS and HTTP/2 options and header order set; you add proxies, a trust store and timeouts, then build it. ImpersonationProfile::apply writes the identity headers of one request. CustomProfileBuilder layers your own JA3, JA4_r or JA4_ro, peetprint, Akamai string and header set over a measured base, and refuses what the stack cannot send before a client exists. On top of it, HttpSession sends requests with that client, keeps an RFC 6265 cookie jar (with the public suffix list), follows redirects by hand, enforces limits (time, body size, redirects, requests per window and site) and detects challenges on every reply. It does not run JavaScript or solve challenges. A request body is held in memory; the reply is read as a stream and capped.

Dual-mode sessions#

StealthSession holds one StealthIdentity (the fingerprint with its locale, the measured browser, the cookies, the client hints, and the egress they were earned through) and works in one of two modes. In HTTP mode an HttpSession sends the requests. In browser mode a real Chrome sends them behind the MitmProxy, so both modes show the same handshake and a clearance earned in one is valid in the other. When a challenge appears in HTTP mode the session escalates to the browser, the browser solves it, and the cookies move into the identity; the session then goes back to HTTP (SessionPolicy decides, and a clearance about to expire sends it back to the browser). The proxy and the HTTP client use the same proxy egress. A Chrome of another major version than the measurement is refused before it loads a page.

The CDP client#

The pure CDP engine of seleniumbase-rs (sb_cdp) starts Chrome itself with --remote-debugging-port (a port the system picks unless you set one) and talks to it over one WebSocket for all tabs; it does not use a pipe. It is async (tokio). The launch arguments contain no --enable-automation. For each tab it sends Page.enable and Emulation.setFocusEmulationEnabled; Network and Fetch are switched on only for features that need them (ad blocking, proxy authentication, request interception, HAR, a fingerprint). It never switches on Runtime, DOM, Log, Debugger or Profiler, because a page can feel that traffic; a test in seleniumbase-rs (tests/sb_cdp_domain_hygiene.rs) fails if a new call does. Elements are found by running document.querySelector through Runtime.evaluate. The scrape crate adds only Network.

Stealth overrides in the page#

A set of built-in evasion providers (seleniumbase_rs::stealth::providers) writes one script that runs before any page script (Page.addScriptToEvaluateOnNewDocument), applied by Page::apply_fingerprint or LaunchOptionsBuilder::fingerprint. Replaced methods go on their prototype and replaced getters on the prototype that owns them, so Object.getOwnPropertyNames is unchanged. Function.prototype.toString is masked (NativeToStringProvider) so patched functions report [native code]. Canvas noise (toDataURL, toBlob, getImageData) and audio noise (AudioBuffer.getChannelData, copyFromChannel, AnalyserNode.getFloatFrequencyData) are a function of the fingerprint seed and the position, so reading the same canvas twice gives the same bytes. Other providers cover WebGL, navigator, plugins, permissions, WebRTC, time zone and locale, geolocation and more.

Challenge detection and mitigation#

challenge::detect reads a response and returns a Detection with a ChallengeKind (None, JsChallenge, Interactive, Blocked, RateLimited) and evidence naming the Vendor (Cloudflare, DataDome, PerimeterX, Akamai, Imperva, Kasada, AWS WAF). A header, a widget or a bare 403 is evidence only; the verdict needs an interstitial marker. MitigationPolicy::decide is a pure function that turns a detection and the attempt context into one Action: Proceed, Retry, Escalate, Solve, RotateEgress, RotateProfile, Cooldown or GiveUp. The defaults are 3 attempts per URL, a new egress after 2 challenges in a row, a new profile after 3, a 900 second cooldown after a block, 60 seconds after a rate limit without Retry-After, and a Retry-After capped at 3600 seconds, with RetryPolicy::network() for backoff. BrowserScraper::solve_challenge is the solving step: for a challenge with a widget it moves the pointer onto the page and calls Page::solve_captcha (a click or a drag on Turnstile, reCAPTCHA, hCaptcha, Friendly Captcha or the DataDome slider; the site decides whether it is accepted), then waits for the vendor’s clearance cookie or for the page to stop being a challenge, bounded by a timeout. A script challenge is only waited for. There is no connection to a solving service.

Proxy pool, rotation and locale from the proxy’s country#

ProxyPool holds (proxy, country) pairs and picks by RotationStrategy (RoundRobin, Random, LeastFailures, StickyPerSite), quarantining a proxy after repeated failures. Changing egress opens a new browser context behind the new proxy (ContextProxySwitch) and drops the old one with its cookies, so the browser is not relaunched. That holds when the new proxy is in a country whose language and time zone the profile already presents; for another country the browser is started again, because the fingerprint is written into every document before its scripts run. The country comes from the proxy spec, and StaticGeoResolver maps it through a built-in table of locales (no network lookup): language list, Accept-Language, time zone and a position near the city, derived from the fingerprint seed.

A built Fingerprint is never changed (fingerprint API v2), so moving one to a locale builds the next one. LocaleExt::localize and EgressLocale::localize return the fingerprint in that locale (language list, Accept-Language, time zone and a position near the city, from the fingerprint’s seed) and leave the one they were given as it was. Every other setting is kept, and a fingerprint that denies geolocation (Spoof::Off) keeps denying it.

use seleniumbase_rs::stealth::fingerprint::{Fingerprint, OsType};use seleniumbase_rs_scrape::egress::geo::{Locale, LocaleExt};# fn demo() -> Result<(), Box<dyn std::error::Error>> {let fingerprint = Fingerprint::randomized_with_major(OsType::Macos, 7, 155);let german = Locale::by_tag("de-DE").ok_or("no such locale")?;let moved = german.localize(&fingerprint)?;assert_eq!(moved.timezone().as_str(), Some("Europe/Berlin"));# Ok(())# }

Profile and identity rotation#

BrowserScraper::rotate_profile draws a new fingerprint from a derived seed for the same operating system, in the locale of the current egress, and restarts the browser with it (cookies start empty; what is known of each site stays). export_identity and import_identity move a StealthIdentity out of and into a scraper; an identity is only accepted through the egress it names, and for the same Chrome version. The identity file is written atomically with mode 0600 and is refused on read if others can access it.

Session state behind a store trait#

Per-site state (successes, challenges, cooldowns) is saved through the StateStore trait. Two implementations ship: InMemoryStateStore and JsonStateStore, one file written to a temporary sibling, synced and renamed over the target, with owner-only permissions. There is no other backend. The JSON store keeps the whole table in memory and rewrites the file on each change, so it suits a few thousand sites at most, and one process may write a file at a time.

Observability#

The scraper reports typed events (ScraperEvent: request, challenge detected and solved, action chosen, mode changed, egress and profile rotated, clearance expiring, cooldown started, identity exported and restored) to an EventSink. URLs and proxies are redacted by construction. Three sinks ship: NoopEventSink, CollectingEventSink and TracingEventSink, which writes to the tracing crate (not log). The proxy also keeps counters (MitmStats).

Human-like input#

Page::human (in seleniumbase-rs, not in these crates) plays a Humanizer plan on a page: the pointer follows a curved, cubic Bézier stroke with a bell-shaped speed profile (minimum jerk), a movement time from Fitts’s law, a faint hand tremor, and sometimes an overshoot and correction. The click lands at a point normally distributed inside the element. Typing has Gaussian intervals around a words-per-minute rate, a slowly drifting rhythm, longer pauses after spaces and punctuation, rare hesitations and, if you ask for it, typos that are noticed and corrected. Scrolling uses wheel notches eased at both ends. With a seed the plans are reproducible. BrowserScraper uses it only for a short pointer placement and scroll before it attempts a widget. Pointer drift while idle is not built.

Code quality#

All three crates declare #![forbid(unsafe_code)], and their error types are thiserror enums.

TLS stack and licences#

mimic uses wreq =6.0.0-rc.31 (Apache-2.0) over btls and tokio-btls =0.5.6, which build BoringSSL (btls-sys) and need cmake and a C/C++ toolchain. The core crate uses reqwest with rustls and ring for its own HTTP (the WebDriver engine, downloads), and the CDP engine talks to Chrome over a plain WebSocket; codec and mimic also use ring for hashing. The three stealth crates are MIT (workspace Cargo.toml and LICENSE); BoringSSL carries its own licences.

Which crate to depend on#

[dependencies]# No BoringSSL and no cmake. Adds the `psl` crate (the Public Suffix List) and# `ring` (through `codec`, for hashing), which the core already builds.seleniumbase-rs = { git = "https://github.com/MustCodeAl/seleniumbase-rs" }seleniumbase-rs-scrape = { git = "https://github.com/MustCodeAl/seleniumbase-rs-stealth" }

[dependencies]# Also builds BoringSSL through btls and drives it with wreq. Needs cmake and a# C/C++ toolchain. It depends on `scrape`.seleniumbase-rs = { git = "https://github.com/MustCodeAl/seleniumbase-rs" }seleniumbase-rs-mimic = { git = "https://github.com/MustCodeAl/seleniumbase-rs-stealth" }

Nothing here is a Cargo feature of seleniumbase-rs any more: you pick a crate. The main types are re-exported at the root of each crate (seleniumbase_rs_scrape::BrowserScraper, seleniumbase_rs_mimic::HttpSession, …); the examples below use the module paths where the module says more. The TLS and HTTP/2 spec error enums are both called SpecError in their modules, so the re-exports are TlsSpecError and H2SpecError. The crate links only one BoringSSL, so do not add a second crate that links it to the same build.

The three crates#

The stealth layer lives in its own repository, seleniumbase-rs-stealth, as three crates:

mimic depends on scrape, which depends on codec; all three depend on seleniumbase-rs. These crates need some core items to be public (in seleniumbase_rs::sb_cdp and seleniumbase_rs::stealth::randomize).

Building the workspace#

The workspace builds against a checkout of seleniumbase-rs next to it: the [patch] section of Cargo.toml points the GitHub dependency at ../seleniumbase-rs.

git clone https://github.com/MustCodeAl/seleniumbase-rsgit clone https://github.com/MustCodeAl/seleniumbase-rs-stealthcd seleniumbase-rs-stealth

cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspace

The first build of mimic compiles BoringSSL (about 1 GB of target directory). Tests that start a real Chrome are the *_chrome.rs files and are #[ignore]d; run them one binary at a time with -- --ignored --test-threads=1. The examples, what each needs, and how to run them are on Runnable examples. The repository is on GitHub:

How the pieces layer#

An arrow reads “is used by”. The scrape crate has no BoringSSL; mimic depends on it and adds BoringSSL. StealthSession moves between HttpSession and a real Chrome behind MitmProxy.

How the crates and sessions layer

Where to go next#