Stealth parity plan: stealthscraper-rs features, JA3/JA4 and Akamai impersonation

Moved: on 2026-10-15 (version 0.3.0) this code moved to seleniumbase-rs-stealth (crates seleniumbase-rs-wire, -scrape, -impersonate); this plan is kept as design history, with the paths the code had here.

Status (2026-10-11, branch feat/pure-cdp-and-mcp-parity at 4184e74): WP0 to WP17 are merged, including WP14 (HttpSession), WP15 (MitmProxy) and WP16 (StealthSession). The WP18 reviews (two passes) are done and their fixes are in progress. WP19 (re-exports and documentation) is in progress. The plan was written on 2026-10-10 as design only, at 8c1394d. Where the code differs from what is written below, the next section says how, and the code wins. Sections 3.11 to 3.13 describe what was built. The user-facing description of what exists is Stealth Scraping; the layout rules are recorded in ADR 003. Sources: the project memory (stealthscraper-feature-parity, project-priorities-2026-10-10), three Context7 reads of /ypno/stealthscraper-rs, the wreq 6.0.0-rc.31, wreq-proto 0.2.6, http2 0.5.20 and btls 0.5.6 sources in ~/.cargo/registry, and our code read through TokenSave.

Status and what the code changed#

Deviations from the plan that the code made. The plan’s text above was not rewritten; read it with these:

• SpecError is split per module. The plan has one SpecError that holds every variant. In the code wire::tls_spec::SpecError holds the TLS variants, wire::h2_spec::SpecError the HTTP/2 variants, and the variant that names an unmeasured browser and system, NotMeasured { kind, os }, lives in wire::emulation::EmulationError (with VersionNotMeasured, Parse, TooLarge, Inconsistent, Headers and Duplicate). wire::emulation::SpecError is an alias for EmulationError, so the plan’s SpecError::NotMeasured is reachable under that name. ImpersonateError wraps each of them as Tls, Http2 and Emulation, next to Ja3, Ja4, Peetprint, Akamai, Headers, Convert, VersionMismatch, Incomplete, Fingerprint, UnknownHeader and Build; there is no single Spec variant.
• KnownGap lives in wire::tls_spec, not in emulation. wire::emulation re-exports it. The plan’s name for the HTTP/2 spec, Http2Spec, is an alias of H2Spec.
• BrowserKind has no serde. The enum in wire/mod.rs derives only Debug, Clone, Copy, PartialEq and Eq. EmulationRef in scrape::passport::identity serializes it through a private module that knows the one spelling, chrome, so a new variant breaks the build there until it is handled.
• Mode lives in scrape::challenge::mitigation. scrape::ledger::state and scrape::passport::policy import it from there.
• Chrome 155 sends a GREASE signature algorithm that the vendored BoringSSL does not write. It is not a KnownGap variant (those are TrustAnchorsExtension and MlDsaSignatureAlgorithms). JA3 and JA4 strip GREASE, so only the peetprint shows it, and a wire test pins the difference and fails once a newer BoringSSL closes it. A TlsSpec read from a hello drops GREASE signature algorithms.
• ScrapeError has three more variants than §3.13 lists: CoolingDown, Navigation and NoEgressAvailable. BrowserScraperBuilder has more settings than the sketch: mitigation, solve_timeout, clock, launcher (the BrowserLauncher port), fingerprint and session_store.
• impersonate also enables brotli, for the certificate compressor of the impersonated hello (Chrome offers brotli certificate compression).
• StealthSession::fetch returns StealthFetch, not Fetched. Fetched is HttpSession::fetch’s { reply, verdict }. The HttpSession surface is larger than the sketch: HttpOptions, Attempt, Rules, Verdict, assess and preflight, and HttpError has more variants (§3.11).
• MitmError is not the sketch’s. It has Bind, Tls, InvalidOptions, Random, Upstream, Egress, Unauthorized, BadTarget, NotIntercepted, HostMismatch, BodyTooLarge, VersionMismatch, UpstreamRequest, ForbiddenDestination and MalformedUpgrade; the SNI mismatch is TlsServerError::SniMismatch (§3.12).
• The MITM proxy relays WebSocket (wss:// only) with the measured WebSocket hello and refuses every other upgrade with an empty 502 (§3.12); it also refuses a request whose user agent claims another Chrome major than the profile’s.
• The SPKI-list trust is measured. What --ignore-certificate-errors-spki-list accepts was read from the Chromium source; the acceptance of a chain that carries the pinned leaf key is now measured on Chrome 155 (headless, macOS) by without_the_proxy_the_leaf_pin_alone_trusts_a_chain_carrying_the_leaf_key in tests/impersonate_mitm_chrome.rs, and the proxy never serves such a chain to a page. Each skipped check is still as read in the source.
• StealthSession defaults to 4 attempts per fetch and starts the MITM proxy lazily (§3.13).
• Re-exports. The main types are re-exported at the top of wire, scrape and impersonate and at the crate root; the two spec SpecErrors are TlsSpecError and H2SpecError. The stealth layer is experimental and not a stable API.

0. Decisions#

1. Nothing from stealthscraper-rs, impersonate-rs, impcurl, curl-impersonate-cli or wreq-util enters Cargo.toml, optional or not. Their numbers (Chrome 153, Safari 27) are hints.
2. Two new features; the default build gains no dependency and no C code. scrape (pure Rust, adds only psl): domain layer, JSON state store, CDP browser leg, TLS and client-hint capture. impersonate = ["scrape", ...]: wreq =6.0.0-rc.31 (BoringSSL through btls, needs cmake and a C/C++ toolchain), MITM proxy, HttpSession, StealthSession, HTTP/2 capture.
3. Every TLS, HTTP/2, header and client-hint value ships as JSON written by our capture tools against the local browsers, embedded with include_str!, parsed once, never typed by hand. An unmeasured browser/OS pair is an error (SpecError::NotMeasured), never a fallback to bare wreq.
4. Custom JA3 and Akamai strings overlay a measured base. What BoringSSL or wreq’s HTTP/2 stack cannot emit is refused with a named SpecError before a client exists; what is accepted is proven by a wire round trip.
5. Hexagonal: src/scrape/domain is pure (no tokio, std::fs/std::net/std::process or clock reads; functions take now), enforced by a syn test that allows from sb_cdp only the data types Proxy, Cookie, SameSite, Captcha and the sync::locked helper. Adapters: src/scrape/adapters (std, tokio, sb_cdp) and src/impersonate (wreq, btls, hyper). Ports: StateStore, EventSink, GeoResolver, EgressSwitch.
6. One thiserror enum per module, mapped to SeleniumBaseError::Stealth { reason } at the crate boundary. tracing only, with redacted fields (instrument(skip_all, fields(url = %redacted))).
7. New modules: #![forbid(unsafe_code)], the pedantic lint block src/lib.rs gives sb_cdp, files under 800 lines, functions under 50, named constants, .expect() only with a reason.

1. Feature map#

2. Layout, features, dependencies#

Layout revised (2026-10-11). The tree below was first written by hexagonal layer (scrape/domain, scrape/adapters). It is now laid out by concern. Work packages, file owners and the contents of each file are unchanged; only the paths move:

• src/scrape/domain/wire/** -> src/wire/** (pure; cfg(any(feature = "scrape", feature = "impersonate"))): ja3, ja4, akamai, ids, tls_spec, h2_spec, h2_frames, header_spec, client_hello, emulation; BrowserKind stays in wire/mod.rs.
• src/scrape/domain/{challenge,markers,mitigation}.rs -> src/scrape/challenge/{detect,markers,mitigation}.rs.
• src/scrape/domain/{egress,proxy_pool,geo,redact}.rs -> src/scrape/egress/{route,pool,geo,redact}.rs.
• src/scrape/domain/{cookie_jar,set_cookie}.rs -> src/scrape/jar/{store,set_cookie}.rs.
• src/scrape/domain/{state,events}.rs -> src/scrape/ledger/{state,events}.rs.
• src/scrape/domain/{identity,session_policy}.rs -> src/scrape/passport/{identity,policy}.rs.
• src/scrape/adapters/** -> src/scrape/runtime/** (everything that does I/O).
• src/impersonate/** stays where it is.

The pure directories checked by tests/scrape_domain_purity.rs are src/wire and src/scrape/{challenge,egress,jar,ledger,passport}. Module names drop “domain”, “adapters” and “ports”; the public type names in §3 are unchanged. The paths in §2.1 below and in §5 read as the old names until this note is applied; the map above is the authority.

2.1 Files and owners (WP = work package, §5)#

src/lib.rs (mod lines, lint block: WP0; re-exports: WP19)   src/sb_cdp/mod.rs (`pub(crate) mod sync`): WP0src/sb_cdp/{context,browser}.rs: WP13, only to port the worktree's ContextRoutesrc/stealth/{randomize,fingerprint}.rs: WP1, additive (LOCALES source, randomized_with_major,    ClientHints.wow64 / form_factors with serde default)src/stealth/cdp.rs, src/stealth/providers/{mod,builtin}.rs, src/sb_cdp/identity.rs: WP17src/scrape/  (feature "scrape")      mod.rs WP0 (re-exports WP19), error.rs WP13 (ScrapeError)  domain/    (pure, enforced)        mod.rs WP0    redact egress geo proxy_pool                                WP1    challenge markers mitigation (+ tests/fixtures/challenge)   WP2    state WP3 | cookie_jar set_cookie WP4 | identity session_policy events WP8    wire/  mod.rs (BrowserKind) WP0 | ids WP5 (flags fixed by WP12's probe)           client_hello ja3 ja4 tls_spec WP6 | akamai h2_frames h2_spec header_spec WP7           emulation WP11 | measured/<browser>-<major>-<os>.json WP9 (TLS, hints), WP11 (H2, headers)  adapters/  mod.rs WP0 | json_store WP3 | tracing_sink identity_file WP8 | capture WP9             cdp browser_scraper egress_switch WP13src/impersonate/ (feature "impersonate")  mod.rs WP0 (re-exports WP19)  tls_server WP10 | capture_h2 WP11 | convert client WP12 | http_session WP14  mitm/{mod,connect,forward,upstream} WP15 | stealth_session WP16examples: capture_tls, capture_hints WP9 | capture_h2 WP11tests: scrape_domain_purity WP0 | fixtures/measured/*.bin WP9, WP11 | impersonate_wire WP12       scrape_browser WP13 | impersonate_http WP14 | impersonate_mitm WP15 | stealth_session WP16       sb_cdp_domain_hygiene, stealth_audit_chrome, stealth_audit_network WP17Cargo.toml, Cargo.lock, justfile, .github/workflows/ci.yml: WP0 onlydocs/tutorials/stealth_scraping.md (new), fingerprint_stealth.md, ROADMAP.md: WP19

WP0 creates every listed source file with only a //! line, so later packages fill files they own and never edit a shared mod.rs. Unit tests stay in-file; integration tests are grouped into the few binaries above because each test binary links the whole crate (disk).

2.2 Cargo.toml (only WP0 edits it)#

Reused, already present: tokio (net, io-util, fs, sync, time, macros), serde, serde_json, thiserror 2, tracing, async-trait, url 2.5, chrono, ring 0.17 (SHA-256 for JA4, SystemRandom for tokens), base64 0.23 (SPKI pins), futures-util; dev: tokio test-util, syn 3. hyper 1.12.0, hyper-util 0.1.21, http-body-util 0.1.5, http 1.5.0 and bytes 1.12.1 are already in Cargo.lock through reqwest.

[dependencies]psl = { version = "2.1.241", optional = true }wreq = { version = "=6.0.0-rc.31", optional = true, default-features = false, features = [    "tokio-rt", "webpki-roots", "gzip", "deflate", "brotli", "zstd", "socks", "stream"] }btls = { version = "=0.5.6", optional = true }        # the BoringSSL wreq links; links = "boringssl" allows onetokio-btls = { version = "=0.5.6", optional = true }hyper = { version = "1.12", optional = true, default-features = false, features = ["server", "http1"] }hyper-util = { version = "0.1.21", optional = true, default-features = false, features = ["tokio"] }http-body-util = { version = "0.1.5", optional = true }http = { version = "1.5", optional = true }bytes = { version = "1.12", optional = true }[features]scrape = ["dep:psl"]impersonate = ["scrape", "dep:wreq", "dep:btls", "dep:tokio-btls", "dep:hyper", "dep:hyper-util",               "dep:http-body-util", "dep:http", "dep:bytes"]# [[example]] capture_tls, capture_hints: required-features ["scrape"]; capture_h2: ["impersonate"]

Not enabled on purpose: wreq cookies (our RFC 6265 jar writes Cookie; redirects are followed by hand), ws, hickory-dns, system-proxy. zstd pulls zstd-sys (C, built by cc), reachable only under impersonate. Licenses: wreq and btls Apache-2.0, btls-sys MIT, tokio-btls MIT OR Apache-2.0, psl MIT/Apache-2.0, all inside deny.toml’s allow-list (its [graph] all-features sees impersonate).

3. Public API#

Conventions for the blocks below: public enums are #[non_exhaustive]; data types derive Clone, Debug, PartialEq and Serialize, Deserialize when persisted; errors derive thiserror::Error; the Debug of anything holding a secret is hand-written and prints <redacted> or counts.

3.1 Redaction and egress (domain/{redact,egress}.rs, WP1)#

pub struct Secret(String);                        // Display/Debug "<redacted>", never Serializeimpl Secret { pub fn new(v: impl Into<String>) -> Self; pub fn expose(&self) -> &str; }pub struct RedactedUrl(String);                   // Serialize; the only constructor redactsimpl RedactedUrl { pub fn new(raw: &str) -> Self; pub fn as_str(&self) -> &str; }/// Values of these query keys become "***"; userinfo and fragment are always dropped.pub const SENSITIVE_QUERY_KEYS: [&str; 12] = ["token", "access_token", "key", "api_key", "sig",    "signature", "password", "pass", "auth", "session", "sid", "secret"];pub struct EgressKey { pub scheme: String, pub host: String, pub port: u16, pub country: Option<CountryCode> }pub struct EgressRef { proxy: sb_cdp::Proxy, key: EgressKey }   // Debug/Display = key; never Serializeimpl EgressRef {    /// http, https, socks5, socks5h; `user:password@` leaves the string here, in the constructor.    pub fn parse(spec: &str, country: Option<CountryCode>) -> Result<Self, EgressError>;    pub fn key(&self) -> &EgressKey;    pub fn proxy(&self) -> &sb_cdp::Proxy;        // ContextOptions::proxy    pub(crate) fn launch_spec(&self) -> Secret;   // LaunchOptionsBuilder::proxy, MITM upstream}pub enum EgressError { Scheme(String), Address(RedactedUrl) }

Only EgressKey is ever serialized (state, identity, events); credentials never are.

3.2 Geo and proxy pool (domain/{geo,proxy_pool}.rs, WP1)#

#[serde(try_from = "String", into = "String")]pub struct CountryCode([u8; 2]);                  // ISO 3166-1 alpha-2, upper caseimpl CountryCode { pub fn new(code: &str) -> Result<Self, GeoError>; pub fn as_str(&self) -> &str; }pub struct Locale { pub tag: &'static str, pub country: &'static str, pub languages: &'static [&'static str],                    pub timezone: &'static str, pub latitude: f64, pub longitude: f64 }pub static LOCALES: &[Locale];                    // superset of randomize::REGIONS, which now derives from itimpl Locale {    pub fn for_country(country: CountryCode) -> Option<&'static Locale>;  pub fn by_tag(tag: &str) -> Option<&'static Locale>;    pub fn accept_language(&self) -> String;      // format checked against Chrome's measured header    pub fn apply_to(&self, fp: &mut Fingerprint); } // locale, languages, accept_languages, timezone, lat/longpub struct GeoInfo { pub country: CountryCode, pub timezone: Option<String>, pub coordinates: Option<(f64, f64)> }#[async_trait] pub trait GeoResolver: Send + Sync {    async fn resolve(&self, egress: &EgressRef) -> Result<GeoInfo, GeoError>; }pub struct StaticGeoResolver;                     // country given with the proxy; no network lookup shipspub enum GeoError { InvalidCountry(String), NoLocale(CountryCode), Unknown(EgressKey) }pub enum RotationStrategy { RoundRobin, Random { seed: u64 }, LeastFailures, StickyPerSite }pub enum EgressOutcome { Success, ConnectError, Challenged, Blocked }pub struct ProxyPool { /* Mutex<PoolState>, locked through sb_cdp::sync::locked */ }impl ProxyPool {    pub fn new(egresses: Vec<EgressRef>, strategy: RotationStrategy) -> Self;    pub fn with_geo_proxies<I, S>(items: I, strategy: RotationStrategy) -> Result<Self, EgressError>        where I: IntoIterator<Item = (S, CountryCode)>, S: AsRef<str>;    pub fn quarantine(self, after_failures: u32, for_secs: u64) -> Self;   // defaults: named consts 3, 600    pub fn pick(&self, site: Option<&str>, now: u64) -> Option<EgressRef>;    pub fn pick_in(&self, country: CountryCode, site: Option<&str>, now: u64) -> Option<EgressRef>;    pub fn pick_other(&self, current: &EgressKey, site: Option<&str>, now: u64) -> Option<EgressRef>;    pub fn report(&self, egress: &EgressKey, outcome: EgressOutcome, now: u64); }

Fingerprint::randomized_with_major(os, seed, major) is randomized with the Chromium major fixed: an identity riding the measured TLS must claim the measured version, never 153 while Chrome 155 renders.

3.3 Challenge and mitigation (domain/{challenge,markers,mitigation}.rs, WP2)#

pub const MAX_SCAN_BYTES: usize = 256 * 1024;pub struct ResponseView<'a> { pub url: &'a str, pub status: u16, pub headers: &'a [(String, String)], pub body: &'a str }pub enum Vendor { Cloudflare, DataDome, PerimeterX, Akamai, Imperva, Kasada, AwsWaf }pub enum ChallengeKind { None, JsChallenge, Interactive, Blocked, RateLimited { retry_after_secs: Option<u64> } }pub struct Evidence { pub vendor: Option<Vendor>, pub markers: Vec<&'static str>, pub widget: Option<sb_cdp::Captcha> }pub struct Detection { pub kind: ChallengeKind, pub evidence: Evidence }pub fn detect(view: &ResponseView<'_>) -> Detection;pub fn clearance_cookie(vendor: Vendor) -> Option<&'static str>;   // cf_clearance, datadome, ...pub enum Mode { Browser, Http }pub struct AttemptContext { pub attempt: u32, pub mode: Mode, pub consecutive_challenges: u32,                            pub can_rotate_egress: bool, pub can_escalate: bool, pub now: u64 }pub enum Action { Proceed, Retry { after: Duration }, Escalate, Solve, RotateEgress, RotateProfile,                  Cooldown { until: u64 }, GiveUp(GiveUpReason) }pub enum GiveUpReason { AttemptsExhausted, Blocked, NoEgressLeft }pub struct MitigationPolicy { pub max_challenge_attempts: u32, pub backoff: RetryPolicy,    pub rotate_egress_after: u32, pub rotate_profile_after: u32, pub block_cooldown_secs: u64 }impl Default for MitigationPolicy { /* 3, RetryPolicy::network(), 2, 3, 900, as named consts */ }impl MitigationPolicy { pub fn decide(&self, d: &Detection, ctx: &AttemptContext) -> Action; }

A challenge needs interstitial markers (just a moment, cf_chl_opt, /cdn-cgi/challenge-platform/h/{b,g}/orchestrate, form/stage ids, cf-mitigated: challenge, per-vendor equivalents: one named const each, with the fixture that proves it). A bare cf-ray, server: cloudflare or Turnstile widget gives None with the evidence attached. Decisions: None -> Proceed; RateLimited -> Cooldown (Retry-After); JS/interactive in Http -> Escalate, in Browser -> Solve, then RotateEgress, then RotateProfile as consecutive_challenges crosses thresholds; Blocked -> RotateEgress else Cooldown; attempts exhausted -> GiveUp.

3.4 State (domain/state.rs, adapters/json_store.rs, WP3)#

pub struct DomainState { pub site: String, pub successes: u64, pub challenges: u64, pub blocks: u64,    pub consecutive_challenges: u32, pub last_challenge: Option<ChallengeKind>, pub last_seen: u64,    pub cooldown_until: Option<u64>, pub preferred_mode: Option<Mode>, pub last_egress: Option<EgressKey> }impl DomainState { pub fn record(&mut self, d: &Detection, mode: Mode, now: u64); pub fn in_cooldown(&self, now: u64) -> bool; }#[async_trait] pub trait StateStore: Send + Sync {    async fn get(&self, site: &str) -> Result<Option<DomainState>, StateError>;    async fn put(&self, state: DomainState) -> Result<(), StateError>;    async fn remove(&self, site: &str) -> Result<(), StateError>;    async fn sites(&self) -> Result<Vec<String>, StateError>; }pub struct InMemoryStateStore;                    // Arc<Mutex<BTreeMap<String, DomainState>>>pub struct JsonStateStore;                        // one file { "schema": 1, "sites": { ... } }impl JsonStateStore { pub fn open(path: impl Into<PathBuf>) -> Result<Self, StateError>; }pub(crate) fn atomic_write(path: &Path, bytes: &[u8]) -> std::io::Result<()>;pub enum StateError { Io { path: PathBuf, source: io::Error }, Corrupt { path: PathBuf, source: serde_json::Error }, Schema(u32) }

atomic_write: sibling temp file opened 0o600 (Unix OpenOptionsExt::mode), write_all, sync_all, rename, then sync_all on the parent directory, inside spawn_blocking. A missing file is an empty store; a corrupt one is StateError::Corrupt, never silently reset. One writer process per file (documented; no cross-process lock).

3.5 Cookies, RFC 6265 with psl (domain/{cookie_jar,set_cookie}.rs, WP4)#

pub struct StoredCookie { pub name: String, pub value: String, pub domain: String, pub host_only: bool,    pub path: String, pub expires: Option<u64>, pub secure: bool, pub http_only: bool,    pub same_site: Option<sb_cdp::SameSite>, pub created: u64, pub last_access: u64 }pub struct CookieJar { /* Vec<StoredCookie>; serde; Debug prints counts only */ }pub enum SiteContext { SameSite { top_level: bool }, CrossSite { top_level_get: bool } }pub enum StoreOutcome { Stored, Replaced, Deleted, Ignored(IgnoreReason) }pub enum IgnoreReason { Malformed, DomainMismatch, PublicSuffix, IpLiteralDomain, SecureFromInsecure,                        HostPrefix, SecurePrefix, TooLarge }impl CookieJar {    pub fn store(&mut self, request_url: &Url, set_cookie: &str, now: u64) -> StoreOutcome;          // §5.2, §5.3    pub fn header_for(&mut self, request_url: &Url, ctx: SiteContext, now: u64) -> Option<String>; // §5.4    pub fn from_cdp(cookies: &[sb_cdp::Cookie], now: u64) -> Self;  pub fn to_cdp(&self, now: u64) -> Vec<sb_cdp::Cookie>;    pub fn expiry_of(&self, name: &str, site: &str) -> Option<u64>;  pub fn remove_expired(&mut self, now: u64) -> usize; }pub fn site_of(host: &str) -> Option<String>;     // registrable domain via psl; None for IPs and bare suffixes

Rules, each a named test: §5.2 attributes; §5.1.1 dates (two-digit years, any token order); Max-Age beats Expires, <= 0 deletes, expiry capped at 400 days (6265bis). Domain is lower-cased, loses a leading dot, must domain-match the host (§5.1.3). Public suffix (§5.3 step 5): ignored unless it equals the host, which then gets a host-only cookie. IP hosts: Domain must equal the IP, cookie host-only, no suffix matching. Paths per §5.1.4. 6265bis as Chrome applies it: an insecure origin cannot set or overwrite a Secure cookie; __Secure- needs Secure; __Host- needs Secure, Path=/, no Domain. SameSite per SiteContext; order per §5.4; bounds (named consts) 4096 bytes, 180 per site, 3000 total, least-recently-accessed eviction. CDP: a leading dot in sb_cdp::Cookie::domain means a domain cookie; expires: None is a session cookie.

3.6 Identity, session policy, events (domain/{identity,session_policy,events}.rs, WP8)#

pub const IDENTITY_SCHEMA: u32 = 1;pub struct EmulationRef { pub kind: BrowserKind, pub major: u32, pub os: OsType }pub struct Clearance { pub site: String, pub vendor: Vendor, pub cookie: String, pub expires_at: u64 }pub struct StealthIdentity {                      // serde; Debug: cookie count and egress key only    pub schema: u32, pub fingerprint: Fingerprint, // UA, platform, HW, WebGL, screen, locale, tz, client_hints, seed    pub emulation: EmulationRef, pub cookies: CookieJar,    pub accept_ch: BTreeMap<String, Vec<String>>, // origin -> hints it asked for (Accept-CH)    pub egress: Option<EgressKey>,                // sticky: the IP that earned the clearance    pub clearances: Vec<Clearance>, pub created_at: u64 }impl StealthIdentity {    /// Refuses an incoherent fingerprint (Fingerprint::validate); clears fingerprint.proxy (may hold a password).    pub fn new(fp: Fingerprint, emulation: EmulationRef, egress: Option<EgressKey>, now: u64) -> Result<Self, IdentityError>;    pub fn refresh_clearances(&mut self, now: u64);   // from the jar + challenge::clearance_cookie    pub fn clearance_for(&self, site: &str) -> Option<&Clearance>;    pub fn check_egress(&self, egress: Option<&EgressKey>) -> Result<(), IdentityError>; }pub enum IdentityError { EgressMismatch { expected: Option<EgressKey>, found: Option<EgressKey> }, Schema(u32),                         Incoherent(Vec<String>), EmulationMismatch { identity: u32, measured: u32 } }pub struct SessionPolicy { pub require_clearance_for_http: bool, pub clearance_margin: Duration,                           pub demote_after_solve: bool, pub prefer: Mode }   // false, 120 s, true, Httppub enum Transition { Stay, Escalate(EscalateReason), Demote(DemoteReason) }pub enum EscalateReason { Challenge(ChallengeKind), ClearanceExpiring { remaining_secs: u64 }, NoClearance, SiteNeedsBrowser, Manual }pub enum DemoteReason { ChallengeSolved, Manual }impl SessionPolicy {    pub fn before_fetch(&self, mode: Mode, id: &StealthIdentity, site: &str, st: Option<&DomainState>, now: u64) -> Transition;    pub fn after_fetch(&self, mode: Mode, d: &Detection, a: &Action, id: &StealthIdentity) -> Transition; }pub enum ScraperEvent {                           // Serialize    Request { url: RedactedUrl, status: u16, mode: Mode, elapsed_ms: u64 },    ChallengeDetected { url: RedactedUrl, kind: ChallengeKind, vendor: Option<Vendor>, mode: Mode },    ChallengeSolved { url: RedactedUrl, kind: ChallengeKind, attempts: u32, elapsed_ms: u64 },    ActionChosen { site: String, action: Action }, ModeChanged { from: Mode, to: Mode, reason: Transition },    EgressRotated { from: Option<EgressKey>, to: Option<EgressKey> }, ProfileRotated { from_seed: u64, to_seed: u64 },    ClearanceExpiring { site: String, remaining_secs: u64 }, CooldownStarted { site: String, until: u64 },    IdentityExported { cookies: usize }, IdentityRestored { cookies: usize } }pub trait EventSink: Send + Sync { fn emit(&self, event: &ScraperEvent); }   // must not blockpub struct NoopEventSink;  pub struct CollectingEventSink;   // adapters: TracingEventSink

URLs enter events only as RedactedUrl and proxies only as EgressKey, so a secret cannot reach an event by construction. adapters/identity_file.rs saves through atomic_write (0600: the file holds cookies) and loads with a schema check.

3.7 Wire parsers and specs (domain/wire/*, WP5 to WP7)#

pub enum BrowserKind { Chrome }                   // WP11 adds Safari once its capture existspub const MAX_CLIENT_HELLO: usize = 64 * 1024;  pub fn is_grease(id: u16) -> bool;  // 0x?a?a, equal bytespub struct ClientHello { pub legacy_version: u16, pub cipher_suites: Vec<u16>, pub extensions: Vec<Extension>,    pub sni: Option<String>, pub alpn: Vec<Vec<u8>>, /* + supported_versions, supported_groups, key_share_groups,    point_formats, signature_algorithms, psk_modes, cert_compression, alps, record_size_limit, delegated_credentials */ }impl ClientHello { pub fn needed_len(prefix: &[u8]) -> Result<Option<usize>, ClientHelloError>;  // incremental                   pub fn parse(records: &[u8]) -> Result<Self, ClientHelloError>; }           // across recordspub enum ClientHelloError { NotHandshake, NotClientHello(u8), Truncated { need: usize, have: usize }, Length, TooLarge }pub struct Ja3 { pub version: u16, pub ciphers: Vec<u16>, pub extensions: Vec<u16>, pub groups: Vec<u16>, pub point_formats: Vec<u8> }impl Ja3 { pub fn parse(s: &str) -> Result<Self, Ja3Error>;   // strips GREASE ids given in the input           pub fn of(hello: &ClientHello) -> Self; }          // Display: "771,4865-4866-...,0-23-...,29-23-24,0"pub enum Ja3Error { FieldCount(usize), Number { field: Ja3Field, token: String }, Empty(Ja3Field), TooLong }pub struct Ja4 { pub a: String, pub b: String, pub c: String }   // Display "a_b_c"impl Ja4 { pub fn of(hello: &ClientHello) -> Self; pub fn raw(hello: &ClientHello) -> String; // JA4_r           pub fn diff(&self, other: &Ja4) -> Ja4Diff; }pub struct TlsSpec { pub legacy_version: u16, pub cipher_suites: Vec<u16>, pub extensions: Vec<u16>,    pub permute_extensions: bool, pub grease: bool, pub supported_groups: Vec<u16>, pub key_share_groups: Vec<u16>,    pub point_formats: Vec<u8>, pub signature_algorithms: Vec<u16>, pub alpn: Vec<String>, pub alps: Option<Alps>,    /* + delegated_credentials, supported_versions, psk_modes, cert_compression, record_size_limit,       ech_grease, ocsp_stapling, sct, session_ticket, renegotiation */ }impl TlsSpec {    pub fn from_client_hello(hello: &ClientHello) -> Self;          // what the capture tool records    pub fn with_ja3(&self, ja3: &Ja3) -> Result<Self, SpecError>;   // overlay; turns permutation off    pub fn with_signature_algorithms(&self, ids: &[u16]) -> Result<Self, SpecError>;    pub fn check_reproducible(&self) -> Result<(), SpecError>;      // against ids::BORINGSSL_* sets    pub fn without_known_gaps(&self, gaps: &[KnownGap]) -> Self;    pub fn ja3(&self) -> Ja3; }pub enum Pseudo { Method, Authority, Scheme, Path }pub struct PriorityFrame { pub stream_id: u32, pub exclusive: bool, pub depends_on: u32, pub weight: u16 }pub struct Akamai { pub settings: Vec<(u16, u32)>, pub window_update: u32, pub priority_frames: Vec<PriorityFrame>,                    pub pseudo_order: [Pseudo; 4] }       // Display: "1:65536;2:0;...|15663105|0|m,a,s,p"impl Akamai { pub fn parse(s: &str) -> Result<Self, AkamaiError>; }pub enum AkamaiError { Sections(usize), Setting { index: usize, token: String }, DuplicateSetting(u16),                       WindowUpdate(String), PriorityFrame { index: usize, token: String }, PseudoOrder(String) }pub struct HeadersPriority { pub depends_on: u32, pub weight: u16, pub exclusive: bool }pub fn read_preface(bytes: &[u8]) -> Result<Option<H2Preface>, H2CaptureError>;  // None = need more bytespub struct Http2Spec { pub settings: Vec<(u16, u32)>, pub window_update: u32, pub priority_frames: Vec<PriorityFrame>,    pub headers_priority: Option<HeadersPriority>, pub pseudo_order: [Pseudo; 4], pub initial_stream_id: Option<u32> }impl Http2Spec { pub fn with_akamai(&self, a: &Akamai) -> Result<Self, SpecError>;                 pub fn check_reproducible(&self) -> Result<(), SpecError>; pub fn akamai(&self) -> Akamai; }pub enum RequestShape { Document, Fetch }pub enum HeaderValue { Literal(String), UserAgent, SecChUa, SecChUaMobile, SecChUaPlatform, AcceptLanguage, Cookie }pub struct HeaderSpec { pub document: Vec<(String, HeaderValue)>, pub fetch: Vec<(String, HeaderValue)>,                        pub priority_by_dest: Vec<(String, String)> }   // h2-only `priority` per sec-fetch-dest

JA4 per FoxIO’s TLS-client definition (BSD-3-Clause): a = t + highest non-GREASE version + d/i

• two-digit cipher and extension counts + first and last ALPN character; b/c = first 12 hex of SHA-256 (ring::digest) over sorted ciphers / sorted extensions minus SNI and ALPN, _, sigalgs in wire order. JA3 is compared as a string (no MD5 dependency). Pseudo order comes from HPACK static-table name indices (1-7), so no Huffman decoder; header names and values are measured over plaintext h1.

3.8 What BoringSSL and wreq cannot reproduce#

ids.rs holds the IANA names and the capability sets BORINGSSL_CIPHERS, BORINGSSL_CLIENT_EXTENSIONS, BORINGSSL_GROUPS, BORINGSSL_SIGALGS. WP5 seeds them from btls 0.5.6’s constants (ExtensionType, KeyShare, SslSignatureAlgorithm); WP12’s probe test builds a client per id, captures the ClientHello and fails if a set claims an id that does not appear on the wire. The probe, not this table, decides.

A custom JA3 turns extension permutation off, so its JA3 on the wire is exact. Measured Chrome keeps permutation on, as Chrome does: its JA3 changes per connection by design, and JA4 is the target.

pub enum SpecError { LegacyVersion(u16), UnsupportedCipher(u16), Tls13Suites,    UnsupportedExtension { id: u16, reason: &'static str }, DuplicateExtension(u16), UnsupportedGroup(u16),    PointFormats(Vec<u8>), UnsupportedSigalg { id: u16, name: &'static str }, UnsupportedSetting(u16),    SettingValue { id: u16, value: u32 }, SettingAlwaysSent(u16), WindowUpdate(u32),    Priority { stream: u32, reason: &'static str }, NotMeasured { kind: BrowserKind, os: OsType } }

3.9 Measured tables (domain/wire/emulation.rs, WP11)#

pub enum KnownGap { TrustAnchorsExtension, MlDsaSignatureAlgorithms }pub struct Provenance { pub captured_on: String, pub tool: String, pub browser: String, pub known_gaps: Vec<KnownGap> }pub struct Emulation { pub kind: BrowserKind, pub major: u32, pub full_version: String, pub os: OsType,    pub user_agent: String, pub tls: TlsSpec, pub tls_websocket: Option<TlsSpec>, pub http2: Option<Http2Spec>,    pub headers: HeaderSpec, pub client_hints: Option<ClientHints>, pub provenance: Provenance }pub fn measured() -> &'static [Emulation];        // include_str!("measured/*.json"), parsed once (OnceLock)pub fn for_kind(kind: BrowserKind) -> &'static Emulation;   // every variant has an entry (test-enforced)pub fn get(kind: BrowserKind, os: OsType) -> Result<&'static Emulation, SpecError>;

Only the local Mac is measured: Chrome on macOS by code through sb_cdp, Safari on macOS when the user opens the capture URL in it. Windows and Linux entries appear only when someone runs the capture tools there (capture_hints also prints a console snippet for machines without Rust); until then get returns NotMeasured and ImpersonationProfile::random picks measured OSes only. Client hints reuse ClientHints: brand order and GREASE brand checked against randomize::chromium_hints(major), the real full version, the per-OS platformVersion.

3.10 impersonation_client (impersonate/{convert,client}.rs, WP12)#

pub struct ImpersonationProfile { /* emulation; effective TlsSpec, Http2Spec, HeaderSpec; UA, hints, Accept-Language */ }impl ImpersonationProfile {    pub fn measured(kind: BrowserKind) -> Result<Self, ImpersonateError>;    pub fn random(seed: u64) -> Result<Self, ImpersonateError>;     // randomized_with_major    pub fn from_fingerprint(fp: &Fingerprint) -> Result<Self, ImpersonateError>;  // UA major must equal measured    pub fn with_locale(self, locale: &Locale) -> Self;  pub fn custom() -> CustomProfileBuilder; }impl CustomProfileBuilder {    pub fn base(self, kind: BrowserKind) -> Self;                  // fills what JA3/Akamai do not carry    pub fn ja3(self, ja3: &str) -> Result<Self, ImpersonateError>; // Ja3Error or SpecError    pub fn akamai(self, akamai: &str) -> Result<Self, ImpersonateError>;    pub fn signature_algorithms(self, ids: &[u16]) -> Result<Self, ImpersonateError>;    pub fn user_agent(self, ua: impl Into<String>) -> Self;  pub fn build(self) -> Result<ImpersonationProfile, ImpersonateError>; }pub fn impersonation_client(profile: &ImpersonationProfile) -> Result<wreq::ClientBuilder, ImpersonateError>;pub enum ImpersonateError { Ja3(Ja3Error), Akamai(AkamaiError), Spec(SpecError),                            VersionMismatch { browser: u32, measured: u32 }, Build(String) }

convert.rs alone names wreq types: TlsSpec -> wreq::tls::TlsOptions (cipher_list, curves_list, sigalgs_list, extension_permutation, permute_extensions, grease_enabled, key_shares, alpn_protocols, alps_*, certificate_compressors, record_size_limit, enable_ech_grease, enable_ocsp_stapling, enable_signed_cert_timestamps, preserve_tls13_cipher_list); Http2Spec -> wreq::http2::Http2Options (settings_order, headers_pseudo_order, headers_stream_dependency, priorities, initial_stream_id, windows); HeaderSpec -> OrigHeaderMap. UA, Sec-CH-UA*, Accept-Language and Cookie are set per request after emulation. Verification stays on; tests trust a root via tls_cert_store, never tls_verify_hostname(false).

3.11 HttpSession (impersonate/http_session.rs and http_session/*, WP14) — as built#

The sketch below is the code’s public surface at 4184e74. HttpSession is Clone (an Arc over the wreq client, the profile, a jar behind a mutex, the egress, the limits, a per-site pacer, the sink, the clock and the protocol each origin settled on).

impl HttpSession {    pub async fn from_page(page: &Page, egress: Option<EgressRef>, options: HttpOptions) -> Result<Self, HttpError>;    pub fn from_identity(identity: &StealthIdentity, egress: Option<EgressRef>, options: HttpOptions) -> Result<Self, HttpError>;    pub async fn get(&self, url: &str) -> Result<HttpReply, HttpError>;    pub fn request(&self, method: Method, url: &str) -> HttpRequest<'_>;   // .shape() .header() .body() .json() .form() .timeout() .send()    pub async fn sync_cookies_back(&self, page: &Page) -> Result<usize, HttpError>;   // only this session's changes; kept on failure    pub fn export_identity(&self, base: &StealthIdentity) -> StealthIdentity;    // outcome.rs: the replies judged by the mitigation and session policies    pub fn assess(&self, reply: &HttpReply, attempt: &Attempt, rules: &Rules, identity: &StealthIdentity) -> Verdict;    pub async fn fetch(&self, url: &str, attempt: &Attempt, rules: &Rules, identity: &StealthIdentity) -> Result<Fetched, HttpError>;    pub fn preflight(&self, url: &str, identity: &StealthIdentity, state: Option<&DomainState>, rules: &Rules) -> Result<Transition, HttpError>;    pub fn profile(&self) -> &ImpersonationProfile;  pub fn egress(&self) -> Option<&EgressKey>;  pub fn cookie_count(&self) -> usize; }pub struct Attempt { pub number: u32, pub previous_challenges: u32, pub can_rotate_egress: bool, pub can_escalate: bool }pub struct Rules { pub mitigation: MitigationPolicy, pub session: SessionPolicy }pub struct Verdict { pub detection: Detection, pub action: Action, pub transition: Transition }   // .is_clear()pub struct Fetched { pub reply: HttpReply, pub verdict: Verdict }impl HttpReply { status() is_success() url() redacted_url() headers() bytes() text() json::<T>() protocol() redirects() elapsed() detect() }pub struct HttpOptions { /* timeout, connect_timeout, max_body_bytes, max_request_bytes, max_redirects,    follow_redirects, pacing, event_sink, trust(CertStore), resolve, bypass, installed_chrome_major, clock */ }pub enum HttpError { Impersonate(ImpersonateError), VersionMismatch { browser: u32, measured: u32 },    Identity(IdentityError), ReservedHeader(String), InvalidHeader(String), ForbiddenMethod(String),    Url { reason: &'static str }, TooManyRedirects(usize), RedirectRefused { url: RedactedUrl, reason: &'static str },    BodyTooLarge(usize), RequestTooLarge(usize), Request { url: RedactedUrl, kind: &'static str },    Client { kind: &'static str }, Options(&'static str), NotUtf8, Json { kind: &'static str },    Browser(SeleniumBaseError) }

What differs from the first sketch, and what the code does:

• Judging is part of the session. assess, fetch and preflight apply MitigationPolicy::decide, SessionPolicy::before_fetch and after_fetch; the plan’s Fetched for a session is therefore the { reply, verdict } pair of HttpSession::fetch, and the dual-mode session returns StealthFetch (§3.13). HttpOptions, Attempt, Rules and Verdict are new types.
• Redirects by hand (10 hops by default, 30 at most; Set-Cookie stored and Cookie sent per hop; 303 to GET; every header the caller set is dropped on a hop to another origin except Accept and Content-Type). A redirect is followed where it points, a private address included.
• Limits are named constants in http_session::options: total time 30 s (at most 1 h), connect 10 s, reply body 64 MiB after decompression (at most 1 GiB), request body 64 MiB, per-site pacing optional.
• Owned headers. User-Agent, Sec-CH-UA*, Accept-Language, Accept-Encoding, Cookie and priority, and the framing headers, are refused with ReservedHeader when the request is sent.
• Page handoff. from_page reads navigator.userAgent, navigator.languages, Browser.getVersion, Network.getAllCookies and the proxy of the page’s browser context. It refuses another Chrome major (VersionMismatch, judged on both the user agent and Browser.getVersion), a browser that is not Chrome, a user agent that says HeadlessChrome, and a system that was not measured (Impersonate).
• The first request to an origin is written for HTTP/2, so an HTTP/1.1-only origin gets the HTTP/2 priority header once; the session then remembers the origin and omits it. Only the second half is asserted by a test.
• wreq errors are reduced to a fixed word (kind); their text, which can name the URL and the proxy password, is dropped. Credentials in a URL are refused.

3.12 MITM proxy (impersonate/{tls_server/*, mitm/*}, WP10 and WP15) — as built#

The CA (WP10) is in impersonate::tls_server; the proxy (WP15) is in impersonate::mitm.

pub struct EphemeralCa { /* btls PKey + X509, memory only */ }impl EphemeralCa {    pub fn generate() -> Result<Self, TlsServerError>;    pub fn spki_pins(&self) -> [String; 2];        // base64(SHA-256(SPKI)) of the CA key and the shared leaf key    pub fn leaf_spki_pin(&self) -> &str;           // the leaf pin alone: the only pin Chrome gets    pub fn chrome_spki_arg(&self) -> String;       // "--ignore-certificate-errors-spki-list=<leaf pin>"    pub fn leaf_for(&self, host: &url::Host<String>, now: SystemTime) -> Result<Arc<Leaf>, TlsServerError>; }  // LRUpub fn acceptor(alpn: &[&[u8]]) -> Result<btls::ssl::SslAcceptor, TlsServerError>;   // no CA argumentpub async fn accept<S>(acceptor: &SslAcceptor, leaf: &Leaf, stream: S, handshake_timeout: Duration)    -> Result<SslStream<S>, TlsServerError>;#[doc(alias = "TlsSpoofingProxy")] pub struct MitmProxy { /* Arc<Inner>; Clone */ }impl MitmProxy {    pub async fn start(profile: ImpersonationProfile, egress: Option<EgressRef>, options: MitmOptions,                       sink: Arc<dyn EventSink>) -> Result<Self, MitmError>;    pub fn port(&self) -> u16;  pub fn local_addr(&self) -> SocketAddr;  pub async fn shutdown(self, grace: Duration);    pub fn proxy_spec(&self) -> Secret;            // http://sb:<token>@127.0.0.1:<port>; holds the token    pub fn browser_proxy(&self) -> sb_cdp::Proxy;  // the same, parsed; password-free Debug    pub fn chrome_args(&self) -> Vec<String>;      // [leaf pin switch, "--proxy-bypass-list=<-loopback>", "--disable-quic"]    pub fn configure_launch(&self, builder: LaunchOptionsBuilder) -> LaunchOptionsBuilder;  // proxy + chrome_args + priming start page    pub fn prime_url(&self) -> String;             // https://sb-mitm.invalid/<128 random bits>, answered by the proxy    pub fn stats(&self) -> MitmStats;  pub fn egress(&self) -> Option<EgressKey>;    pub fn swap_upstream(&self, egress: Option<EgressRef>) -> Result<(), MitmError>;    pub fn swap_profile(&self, profile: ImpersonationProfile) -> Result<(), MitmError>; }pub struct MitmOptions { /* max_connections, max_header_bytes, max_headers, max_body_bytes, idle_timeout,    handshake_timeout, connect_timeout, read_timeout, websocket_idle_timeout, max_websocket_bytes (u64),    trust_upstream_root(der), resolve(host, addr), ca(Arc<EphemeralCa>), allow_private_destinations(bool) */ }pub enum MitmError { Bind(io::Error), Tls(TlsServerError), InvalidOptions(&'static str), Random,    Upstream(ImpersonateError), Egress(&'static str), Unauthorized, BadTarget(&'static str),    ForbiddenDestination(&'static str), NotIntercepted(&'static str), HostMismatch,    MalformedUpgrade(&'static str), BodyTooLarge { limit: usize },    VersionMismatch { browser: u32, measured: u32 }, UpstreamRequest { url: RedactedUrl, kind: &'static str } }pub struct MitmStats { connections, tunnels, unauthorized, refused, forwarded, upstream_errors, websockets, primed }pub const PRIME_HOST: &str = "sb-mitm.invalid";

SniMismatch of the first sketch is TlsServerError::SniMismatch, reached as MitmError::Tls(..). The variants after Egress are what the proxy logs when it refuses a request; the browser only sees an empty 407, 400, 403, 404 (on the priming host only), 413, 431 or 502.

Differences from the first sketch that matter:

• WebSocket is relayed, as the plan sketched, after WP15 (mitm/websocket.rs). A GET with Upgrade: websocket and Connection: upgrade inside a CONNECT tunnel takes the ordinary checks (token, destination, Host, user agent) and goes upstream over HTTP/1.1 through a second wreq client built from the measured tls_websocket spec minus its known gaps (ALPN http/1.1 only), with the same egress, PublicOnly resolver, roots and timeouts as the ordinary client. On the wire its JA4 is t13d1515h1_8daaf6152771_0a20fe35d3a5: Chrome’s measured t13d1516h1_8daaf6152771_5b37b36a67c0 without the trust-anchors extension and the ML-DSA signature algorithms (the known gaps), which is what the_upstream_hello_is_the_measured_websocket_hello_minus_exactly_the_known_gaps asserts. The browser’s header order is kept without the hop-by-hop fields; the proxy writes Connection: Upgrade and Upgrade: websocket itself, and the handshake fields the measured lists lack get Chrome’s spelling (Sec-WebSocket-Key and the rest). The upstream 101 is relayed with its own headers; the Sec-WebSocket-Accept check is left to the browser. A 101 that switches to another protocol is an empty 502; any other answer reaches the browser as an ordinary answer. The bytes are then copied both ways with tokio::io::copy_bidirectional in the browser connection’s task (so the accept loop’s JoinSet still owns everything), bounded by websocket_idle_timeout (300 s, no byte either way) and max_websocket_bytes per direction (512 MiB; the read that crosses it is not passed on), and both sides are shut down together within one 5 s grace (common::shutdown::cleanup_within) when one closes, a bound is hit or the proxy stops. Every other upgrade is an empty 502; half an upgrade, another method, HTTP/1.0 or a body is an empty 400 (MalformedUpgrade); a handshake outside a tunnel is an empty 502. A custom profile’s WebSocket connections send its base browser’s measured WebSocket hello; a profile without one gets a 502 for every handshake. Tests: tests/impersonate_mitm/websocket.rs (an echo site written in the test, the measured WebSocket JA4 on the wire, the hostile inputs) and, on headless Chrome 155, a_page_opens_a_websocket_through_the_proxy_and_receives_its_echo and a_plain_ws_websocket_goes_through_the_proxy_to_its_site in tests/impersonate_mitm_chrome.rs.
• ws:// is relayed in plain text, the handshake only (mitm/sniff.rs, forward::forward_plain). Measured on Chrome 155 with a recording proxy: a ws:// WebSocket reaches an HTTP proxy as CONNECT host:port (port 80 when the URL names none), never as an absolute-form GET, and the tunnel then carries the handshake in plain text, with the wss:// fields in the same order (Origin is the page’s, null on a blank tab). The proxy reads the first 4 bytes of every tunnel and puts them back: 0x16 (a TLS record) goes to the TLS leg as before; GET is served as HTTP/1.1 with the same head limits, and a WebSocket handshake in it goes through forward exactly as a wss:// one (token at the CONNECT, destination filter at the CONNECT and in the resolver, Host against the target with the http default port, user-agent major, header rewrite, the relay bounds), upstream as http:// on a plain connection, so no ClientHello applies. Any other plain-text request is an empty 502 (400 for half an upgrade) and closes the tunnel; first bytes that are neither close it unanswered and count as refused. This keeps the intercept-only rule: nothing is relayed before the proxy has parsed and checked a WebSocket handshake and the upstream has answered 101, which is also when wss:// bytes start to be copied. tunnels still counts TLS legs only; a ws:// WebSocket counts in websockets. Not tested: a ws:// WebSocket through an egress (the plain handshake leaves the way any http:// request does, so an HTTP egress has to relay the upgrade itself).
• The priming page (closes the limit that a WebSocket could not be a tab’s first request). Measured on Chrome 155 (headless): a WebSocket’s CONNECT that gets the 407 is never surfaced as Fetch.authRequired, so ProxyAuth cannot answer it and Chrome does not retry it; credentials Chrome was given for any tab’s request are cached for the whole browser context and sent preemptively on every later CONNECT, WebSockets included (a navigation in one tab, then a WebSocket as another tab’s first request, goes through); a browser context created afterwards has a cache of its own and does not inherit the default context’s. So the proxy owns https://sb-mitm.invalid/<128 random bits> (PRIME_HOST under the reserved .invalid, MitmProxy::prime_url): after the token check a GET of exactly that is answered by the proxy with an empty, no-store HTML page (MitmStats::primed); anything else for that host, Chrome’s favicon.ico included, is an empty 404; nothing for it goes upstream or to the event sink. configure_launch makes it the start page unless the builder already has one, so Browser::launch loads it through the 407 and ProxyAuth before returning; a context behind the proxy loads it in its first tab. A browser-level answer was tried and is not possible: with Fetch.enable { handleAuthRequests: true } sent to the browser target as well, no Fetch.authRequired (nor any Fetch event) arrives there for the WebSocket’s 407, and the WebSocket still fails (measured on Chrome 155). Measured as well: the default tab’s history.length after the priming page and one navigation is 2, as without it (the start page replaces the initial about:blank). Tests: a_websocket_as_a_new_tabs_first_request_through_the_proxy_gets_its_echo and a_context_behind_the_proxy_takes_the_credentials_from_the_priming_page (Chrome), and the priming tests in tests/impersonate_mitm/access.rs.
• A request whose User-Agent names another Chrome major than the profile’s is refused (MitmError::VersionMismatch), because the measured handshake would contradict the header.
• The first request to an HTTP/1.1-only origin carries the HTTP/2 priority header: the upstream protocol is settled by ALPN after the request was prepared. The proxy remembers the origin and omits the header from then on (asserted: an_http1_origin_gets_the_measured_spelling_and_no_priority_once_known).
• Options and bounds (max_connections 1 to 4096, default 256; request head 8 KiB to 1 MiB, default 64 KiB; headers 1 to 1024, default 128; body default 512 MiB; idle 60 s, handshake 10 s, connect 30 s, read 60 s; WebSocket idle 300 s and 512 MiB per direction) are checked by start.
• StealthSession starts the proxy lazily, the first time a browser is needed (§3.13).

Security rules, each with a WP15 test:

• CA: P-256, critical CA:TRUE, pathlen:0, keyCertSign|cRLSign, random serial, per process, never on disk. Leaves: one shared per-process key, SAN exactly the CONNECT host (DNS or IP via url::Host::parse, no wildcard, max 253 chars), EKU serverAuth, short validity. The server presents the leaf alone (no CA certificate in the chain).
• Chrome gets --ignore-certificate-errors-spki-list=<leaf pin> (honoured only with --user-data-dir, which browser_args always passes), never --ignore-certificate-errors, and never the CA pin. What the switch trusts, read from Chromium’s services/network/ignore_errors_cert_verifier.cc: Chrome hashes the SPKI of every certificate in the presented chain and, when one hash is in the list, accepts the connection with no other check (no signature, host name, validity or revocation). So it trusts any chain that contains a certificate with the pinned public key, from any sender; the key is public (every handshake carries it), so “this CA and its leaves” would be a wrong description. The acceptance is now measured on Chrome 155 (headless, macOS): tests/impersonate_mitm_chrome.rs::without_the_proxy_the_leaf_pin_alone_trusts_a_chain_carrying_the_leaf_key starts Chrome with the pin and no proxy and it loads a page from a loopback server whose chain is its own self-signed certificate followed by the proxy’s leaf. That test does not isolate each skipped check (signature, host name, validity, revocation), which stay as read in the source. Narrowing it is done by pinning only the leaf key and sending the leaf alone, so the CA key and an intermediate are no second way in. The rest is WP15’s duty: every connection of that Chrome profile must end at the MITM (--proxy-server with --proxy-bypass-list=<-loopback> so loopback hosts are intercepted too, --disable-quic), a CONNECT the MITM cannot intercept is refused rather than tunnelled blind, and no leaf certificate is logged or written into a capture artifact. Its real-Chrome test, a_loopback_server_carrying_the_pinned_key_is_never_served_to_the_page: behind the MITM, a loopback server whose chain carries the pinned leaf key is never served to the page (the request goes through the MITM, whose upstream verification refuses it, and the page sees a 502).
• Binds 127.0.0.1:0 only. A random 128-bit token (ring::rand::SystemRandom) is required as Proxy-Authorization: Basic on every CONNECT and absolute-form request (constant-time compare, else 407); Chrome gets it through ProxyAuth, so it is never on Chrome’s command line.
• SNI must equal the CONNECT host or the connection closes. The intercepted leg offers ALPN http/1.1.
• Stripped before replay: Connection and its listed tokens, Keep-Alive, Proxy-Connection, Proxy-Authorization, Proxy-Authenticate, TE (but trailers), Trailer, Transfer-Encoding, Upgrade, and Host (the upstream client writes it). On a WebSocket handshake the proxy writes the Connection: Upgrade and Upgrade: websocket pair itself, in the browser’s places for them. Chrome’s header order is kept; the h2-only priority header comes from the measured table.
• Upstream verification always on (webpki roots, or only the roots added with trust_upstream_root); failure answers an empty 502. The upstream client follows no redirects, decompresses nothing (so Content-Encoding passes through) and sets no default headers: the browser’s own go out.
• Bodies stream both ways, never buffered whole; bounds: 256 connections, 64 KiB request head, named idle and connect timeouts. A relayed WebSocket has its own idle and byte bounds (see above).
• swap_upstream builds the new client first, then swaps an RwLock<Arc<Upstream>>; in-flight requests keep the old one. Logs carry RedactedUrl/EgressKey only; Drop uses shutdown::spawn_cleanup.

3.13 BrowserScraper and StealthSession (WP13, WP16) — StealthSession as built#

The BrowserScraper sketch is the first design; the deviations section at the top lists what the code added (ScrapeError::{CoolingDown, Navigation, NoEgressAvailable} and more builder settings). The StealthSession block further down is the code’s surface at 4184e74.

// scrape::adapters, feature "scrape": CloudScraper parity without TLS spoofingimpl BrowserScraper {                             // Browser, Option<BrowserContext>, Page, pool, store, sink, policy    pub fn builder() -> BrowserScraperBuilder;  pub fn page(&self) -> &Page;    pub async fn fetch(&mut self, url: &str) -> Result<BrowserFetch, ScrapeError>;    pub async fn solve_challenge(&mut self) -> Result<Detection, ScrapeError>;    pub async fn rotate_profile(&mut self) -> Result<(), ScrapeError>;  pub async fn rotate_egress(&mut self) -> Result<(), ScrapeError>;    pub async fn export_identity(&self) -> Result<StealthIdentity, ScrapeError>;    pub async fn import_identity(&mut self, id: &StealthIdentity) -> Result<(), ScrapeError>;    pub async fn close(self) -> Result<(), ScrapeError>; }impl BrowserScraperBuilder {                      // each returns Self unless noted    pub fn launch_options(self, b: LaunchOptionsBuilder) -> Self;    pub fn with_geo_proxies<I, S>(self, p: I) -> Result<Self, ScrapeError> where I: IntoIterator<Item = (S, CountryCode)>, S: AsRef<str>;    pub fn proxy_strategy(self, s: RotationStrategy) -> Self;   pub fn with_max_challenge_attempts(self, n: u32) -> Self;    pub fn with_state_store(self, s: Arc<dyn StateStore>) -> Self; pub fn with_event_sink(self, e: Arc<dyn EventSink>) -> Self;    pub fn geo_resolver(self, g: Arc<dyn GeoResolver>) -> Self; pub fn per_site_rate(self, calls: usize, window: Duration) -> Self;    pub fn seed(self, seed: u64) -> Self;  pub fn egress_switch(self, s: Box<dyn EgressSwitch>) -> Self;    pub async fn build(self) -> Result<BrowserScraper, ScrapeError>; }#[async_trait] pub trait EgressSwitch: Send + Sync {   // port; ContextProxySwitch = new BrowserContext, no relaunch    async fn switch(&self, browser: &Browser, egress: Option<&EgressRef>) -> Result<Page, ScrapeError>; }// adapters::cdp: goto_with_response (Network.responseReceived, main frame), detect_page, apply_locale,// export_cookies (Cookies::all), import_cookies (Cookies::set), wait_for_clearance (poll_until)pub enum ScrapeError { Egress(EgressError), Geo(GeoError), State(StateError), Identity(IdentityError),    Unsolved { url: RedactedUrl, kind: ChallengeKind, attempts: u32 }, GaveUp { url: RedactedUrl, reason: GiveUpReason },    Browser(SeleniumBaseError) }       // StealthError (WP16) = ScrapeError | HttpError | MitmError | ImpersonateError// impersonate, feature "impersonate": dual mode (impersonate::stealth_session and stealth_session/*)impl StealthSession {    pub fn builder() -> StealthSessionBuilder;    pub async fn fetch(&mut self, url: &str) -> Result<StealthFetch, StealthError>;   // not `Fetched`    pub async fn escalate(&mut self) -> Result<(), StealthError>;  pub async fn demote(&mut self) -> Result<(), StealthError>;    pub async fn rotate_profile(&mut self) -> Result<(), StealthError>;  pub async fn rotate_egress(&mut self) -> Result<(), StealthError>;    pub async fn export_identity(&self) -> Result<StealthIdentity, StealthError>;    pub async fn import_identity(&mut self, identity: &StealthIdentity) -> Result<(), StealthError>;    pub fn mode(&self) -> Mode;  pub fn identity(&self) -> &StealthIdentity;  pub fn egress(&self) -> Option<&EgressKey>;    pub fn seed(&self) -> u64;  pub fn http(&self) -> Option<&HttpSession>;  pub fn page(&self) -> Option<&Page>;    pub async fn close(self) -> Result<(), StealthError>; }impl StealthSessionBuilder {   // launch_options, launcher, with_geo_proxies, proxy_strategy, with_max_challenge_attempts,    // mitigation, policy(SessionPolicy), with_state_store, with_event_sink, geo_resolver, per_site_rate, seed,    // fingerprint, restore(identity), http_options(HttpOptions), mitm_options(MitmOptions), solve_timeout, clock    pub async fn build(self) -> Result<StealthSession, StealthError>; }pub struct StealthFetch { pub url: String, pub status: u16, pub headers: Vec<(String, String)>, pub body: String,    pub detection: Detection, pub mode: Mode, pub attempts: u32, pub solved: Option<ChallengeKind>, pub egress: Option<EgressKey> }pub enum StealthError { Scrape(ScrapeError), Http(HttpError), Mitm(MitmError), Impersonate(ImpersonateError) }

Deviations of StealthSession from the sketch: fetch returns StealthFetch (the plan’s Fetched is the { reply, verdict } of HttpSession::fetch); the plan’s .profile() and .mitm() are fingerprint(..) and mitm_options(..); StealthError has no Spec-style variant of its own (Unsolved, GaveUp and CoolingDown arrive as StealthError::Scrape); the attempts default to 4, one more than a browser-only scraper (DEFAULT_ATTEMPTS: one load meets the challenge over HTTP, one solves it in the browser, one uses the clearance over HTTP); and the MITM proxy is started lazily, the first time a browser is needed (at build() when the policy prefers the browser), and lives until close(), so its port and its certificate stay the same across browsers.

fetch: (1) site_of, DomainState, per-site pacing, a site in cooldown is not requested. (2) before_fetch; escalate = start the MITM proxy if it is not running (or point it at the identity’s profile with swap_profile), launch Chrome with the identity’s fingerprint and cookies through configure_launch; demote = take the cookies out of Chrome, HttpSession::from_identity on the same egress, close Chrome. (3) Fetch, detect. (4) decide, then apply Solve, Escalate, RotateEgress (pick_other, swap_upstream, new identity.egress, clearances dropped: they belong to the old IP), RotateProfile, Cooldown, Retry or GiveUp, bounded by max_challenge_attempts. (5) Record state; after a solve, demote if demote_after_solve. Every transition emits ModeChanged. Both transports share one egress; restore refuses an identity with another (EgressMismatch). rotate_profile: new seed, randomized_with_major, Chrome relaunched behind the same MITM port and egress, cookies and clearances dropped, DomainState kept. A launched browser lives until we close it: no idle timeout, never Instant + Duration::MAX, lifetimes are named constants.

3.14 Hardening hooks (WP17)#

stealth::providers::session_registry(fp: &Fingerprint) -> EvasionRegistry: the default registry without PluginsProvider and ConnectionProvider; a test checks the assembled script never names plugins, mimeTypes, pdfViewerEnabled or connection. ClientHintsProvider returns early when !isSecureContext. sb_cdp::identity sends the screen through Emulation.setDeviceMetricsOverride { width: 0, height: 0, screenWidth, screenHeight, ... }. stealth::cdp::enable_default_domains stops sending Runtime.enable.

4. Tests and gates#

4.1 Wire proofs#

• TLS: TlsCapture binds 127.0.0.1:0, reads until ClientHello::needed_len (bounded size and timeout), recomputes JA3 (GREASE removed) and JA4. Clients dial https://capture.test:<port> via wreq ClientBuilder::resolve or Chrome --host-resolver-rules=MAP capture.test 127.0.0.1:<port>, so SNI is real.
• HTTP/2, our client: H2cCapture with http2_only() (prior knowledge, plain TCP) reads the preface, SETTINGS, WINDOW_UPDATE, PRIORITY and first HEADERS, answers SETTINGS + ACK + :status 200 (HPACK 0x88), and computes the Akamai string, HEADERS priority and pseudo order.
• HTTP/2, real browser: capture_h2 does the same behind tls_server::acceptor (ALPN h2); --h1 (ALPN http/1.1) records header names, values and order in plaintext.
• Oracle (ignored, when tshark >= 4.2 is on PATH): captured bytes wrapped in a pcap; tshark’s tls.handshake.ja3_full and tls.handshake.ja4 must equal ours.

4.2 Per module#

#[ignore] tests never run in CI; real-Chrome binaries run one at a time with --test-threads=1.

4.3 Gates (definition of done for every package)#

# lane A (pure Rust). Lane B: CARGO_TARGET_DIR=$SCRATCH/target-imp CMAKE_BUILD_PARALLEL_LEVEL=1, --features impersonateexport CARGO_BUILD_JOBS=1 CARGO_TARGET_DIR=$SCRATCH/target-scrapecargo fmt --all -- --checkcargo clippy --features scrape --profile test --lib -- -D warnings      # lib incl. #[cfg(test)]cargo clippy --features scrape --test <wp_test> --example <wp_example> -- -D warningscargo test --features scrape --lib scrape::          # impersonate:: in lane Bcargo test --features scrape --test <wp_test>cargo test --features scrape --doc scrapecargo llvm-cov --features scrape --lib --summary-only    # domain packages: >= 80 % lines on owned filescargo audit && cargo deny check                      # WP0, any package that changes Cargo.lock, WP18, WP19ripsecrets <owned files>                                 # packages that handle credentials or cookies

Never --all-features (it pulls gui, tui, rustwright); --all-targets clippy runs once, in WP19. Review checklist: owned files only, no unsafe, files < 800 lines, functions < 50, named constants, no secret in any Debug/log/event, tracing::instrument(skip_all) on async pub fns.

5. Work packages#

Models: haiku = mechanical or high-volume, sonnet = primary implementation, opus = security-sensitive design and audits. Size: S up to 400 lines with tests, M 400-900, L 900-1600. Build: check, test, chrome (needs the local browser), B (impersonate lane, BoringSSL). Every package needs a build.

Schedule: at most two builds at once, CARGO_BUILD_JOBS=1, two shared target dirs: target-scrape (lane A) and target-imp (lane B, the only place BoringSSL compiles). Cargo’s dir lock serialises builds per lane, so at most two compilations run at once; real Chrome runs one test binary at a time across both agents (waves 7-8 take turns). Delete both dirs after WP19.

Every work package names its file list; a package that needs a file it does not own stops and reports instead of editing it.

6. Risks, open questions, not copied#

Cannot be done yet with btls 0.5.6 / wreq rc.31:

• Extension 0xca34 (trust anchors) needs btls#168 and wreq#1273 in a release. If the local Chrome sends it, our JA4 differs in extension count and c hash: recorded as KnownGap::TrustAnchorsExtension; the wire test asserts the difference is exactly the known gaps and fails once one disappears. Fix: bump both exact pins in one package, rerun the probe and just measure.
• ML-DSA sigalgs 0x0904/0x0905/0x0906 cannot be emitted: a KnownGap for measured entries, UnsupportedSigalg for custom specs. Experimental/GREASE H2 settings: UnsupportedSetting.
• Chrome permutes extensions per connection, so a preset’s JA3 varies by design (JA4 is stable); a custom JA3 pins the order and matches exactly.

Risks:

• Version drift: once the local Chrome updates past the measured major, HttpSession and the MITM refuse (VersionMismatch) until just measure reruns; an ignored test flags it.
• Only macOS is measured until Windows/Linux captures exist, so other personas cannot use HTTP mode or the MITM. Deliberate: no reasoned values.
• The h1 intercepted leg drops Chrome’s h2-only priority header; the MITM re-adds the measured value. Any other h1/h2 difference WP11 finds must go into HeaderSpec, or it leaks.
• ProxyAuth relies on Fetch interception (WP15 measures the cost); rc pins may change API (convert.rs is the only file naming wreq types).
• BoringSSL needs cmake, a C/C++ toolchain and about 1 GB of target disk, and psl is a large generated crate; WP0 records time, peak memory and disk for both with one job.
• No HTTP/3 in HTTP mode (wreq has none; Chrome behind a proxy skips QUIC too). CHIPS cookies and Critical-CH retries are not modelled. JsonStateStore is single-writer.
• CA key material is freed, not guaranteed wiped (no zeroize in btls): acceptable for an ephemeral, loopback-only CA. A matching fingerprint does not make automation permitted; the tutorial says so.

Questions the plan raised, and the answers (2026-10-10):

1. --remote-debugging-pipe needs fds 3/4 in the child, which std only offers through unsafe pre_exec (command-fds is unsafe inside). Accept that, or keep --remote-debugging-port=0? Answer: keep --remote-debugging-port=0. No unsafe; the pipe transport is deferred.
2. Try ALPN h2 on the intercepted leg (Chrome’s own header block) instead of the recorded http/1.1? Answer: keep http/1.1 on the browser-to-proxy leg.
3. Will you run the capture tools on Windows and Linux machines, and open the URL in Safari in WP11? Answer (user): macOS only for now. Windows, Linux and Safari personas stay refused with SpecError::NotMeasured until they are captured on those systems.
4. v2: one MITM for many BrowserPool contexts, routed by per-context token (ProxyAuth::register)? Answer: later. One proxy per context for now.
5. Accept peetprint or JA4_r strings as further custom inputs (they carry signature algorithms)? Answer (user): “custom inputs are for everything”. Every part of a profile can be supplied by the caller, layered over a measured base: JA3, JA4 and JA4_r, peetprint, Akamai HTTP/2 fingerprints, header names, order and values, the User-Agent, Accept-Language and the client-hint set. Each input is checked for what BoringSSL and wreq can emit; anything they cannot reproduce is refused with a named SpecError before a client exists, and every accepted input is proven by a wire round trip. WP6 and WP7 therefore also parse JA4_r and peetprint (WP6), and the builder in WP12 takes each of them.

Deliberately not copied: their code and names (CloudScraper is BrowserScraper/StealthSession); their measured values (Chrome 153, Safari 27); wreq-util and its presets, including the worktree draft’s CHROME_PROFILES; RedbStateStore and persistence; log sinks (we use tracing); headless_chrome and its idle timeout; the no_std domain check; a crate-wide #![forbid(unsafe_code)] (the new modules get it); --ignore-certificate-errors; any CA key on disk; built-in third-party IP-geolocation (port only); JA4+ beyond JA4 (JA4H, JA4S, JA4X… are under the FoxIO License); automatic planner and reviewer agents on every change.