TLS Server and MitmProxy

The ephemeral-CA TLS server#

mimic::tls_server is the building block under capture_h2 and under MitmProxy. EphemeralCa::generate() creates a P-256 certificate authority in memory for the life of one process. leaf_for(host, now) issues a leaf for one DNS name or IP address (never a wildcard), cached in a count-bounded LRU. acceptor(&[ALPN_HTTP1]) builds the BoringSSL server context (TLS 1.2 and 1.3, no session resumption), and accept runs one handshake and refuses the connection when the client’s SNI names another host.

use std::time::SystemTime;use seleniumbase_rs_mimic::tls_server::{    accept, acceptor, EphemeralCa, ALPN_HTTP1, DEFAULT_HANDSHAKE_TIMEOUT,};# async fn demo(tcp: tokio::net::TcpStream) -> Result<(), Box<dyn std::error::Error>> {let ca = EphemeralCa::generate()?;let chrome_flag = ca.chrome_spki_arg(); // pass to Chrome together with --user-data-dirlet server = acceptor(&[ALPN_HTTP1])?;let host = url::Host::parse("example.com")?;let leaf = ca.leaf_for(&host, SystemTime::now())?;let tls = accept(&server, &leaf, tcp, DEFAULT_HANDSHAKE_TIMEOUT).await?;# drop((chrome_flag, tls));# Ok(())# }

Things to know before you use it:

MitmProxy#

HttpSession covers the requests you make yourself. mimic::mitm::MitmProxy covers the ones a real Chrome makes. It is a loopback forward proxy that ends the browser’s TLS with a leaf of the ephemeral CA and replays each request through the impersonating client, so the origin sees the ClientHello, HTTP/2 start and header spelling of the profile (the measured ones, minus the known gaps) instead of whatever the installed browser would have sent. Both legs then look alike on the wire, and a clearance earned in the browser is valid over HttpSession.

One request through MitmProxy.

use std::sync::Arc;use std::time::Duration;use seleniumbase_rs_mimic::{ImpersonationProfile, MitmOptions, MitmProxy};use seleniumbase_rs::sb_cdp::{Browser, LaunchOptions};use seleniumbase_rs_scrape::NoopEventSink;use seleniumbase_rs_codec::BrowserKind;# async fn demo() -> Result<(), Box<dyn std::error::Error>> {let profile = ImpersonationProfile::measured(BrowserKind::Chrome)?;let mitm =    MitmProxy::start(profile, None, MitmOptions::default(), Arc::new(NoopEventSink)).await?;let options = mitm.configure_launch(LaunchOptions::builder()).build()?;let browser = Browser::launch(options).await?;let page = browser.new_page_at("https://example.com/").await?;# drop(page);browser.close().await?;mitm.shutdown(Duration::from_secs(5)).await;# Ok(())# }

MitmProxy::start(profile, egress, options, sink) binds 127.0.0.1 on a port the system picks. egress is the proxy the replayed requests leave through (or None); sink receives redacted events. configure_launch points a Chrome launch at the proxy and adds the switches it needs.

What the proxy does with each connection#

An absolute-form http:// request is checked the same way and forwarded without CONNECT. A WebSocket handshake inside a tunnel is relayed (see WebSocket below). The browser only ever sees an empty status (407, 400, 403, 404, 413, 431 or 502; the 404 only on the proxy’s priming host); the reason is a MitmError variant in the log (Unauthorized, BadTarget, ForbiddenDestination, NotIntercepted, HostMismatch, MalformedUpgrade, BodyTooLarge, VersionMismatch, UpstreamRequest). start itself can fail with Bind, Tls, InvalidOptions, Random, Upstream or Egress.

What Chrome must be told#

mitm.configure_launch(builder) sets the proxy, makes the proxy’s priming page the start page (see the priming page) and adds mitm.chrome_args():

• the --ignore-certificate-errors-spki-list switch with the leaf pin only (never the CA pin),
• --proxy-bypass-list=<-loopback> so loopback hosts go through the proxy too, and
• --disable-quic.

Because that switch trusts any chain that carries the pinned key (see the CA section), the pin is safe only while every connection of the profile ends at the proxy. Without the last two switches a loopback server or a QUIC connection would reach Chrome directly.

The proxy refuses a request whose User-Agent claims a Chrome major other than the profile’s.

Bounds and options#

MitmOptions sets the bounds; MitmProxy::start refuses a value outside them with MitmError::InvalidOptions.

trust_upstream_root(der) adds a root for upstream verification; once one is added, only the added roots are trusted, not the built-in web roots, and verification is never turned off. resolve(host, addr) pins a host name for the upstream client, and ca(Arc<EphemeralCa>) shares a CA.

Bodies stream both ways; nothing is read whole into memory. mitm.stats() gives counts (connections, tunnels, unauthorized, refused, forwarded, upstream_errors, websockets, primed).

WebSocket#

A page’s wss:// WebSocket reaches the proxy as a CONNECT whose TLS the proxy ends, followed by the opening handshake (GET with Upgrade: websocket and Connection: Upgrade, RFC 6455). The proxy:

Chrome sends no keep-alive of its own on a WebSocket, so a site that pings less often than websocket_idle_timeout loses its idle connections. A custom profile’s WebSocket connections send the measured WebSocket hello of its base browser.

ws:// takes the same road without TLS. Measured on Chrome 155: a page’s ws:// WebSocket reaches an HTTP proxy as a CONNECT (to port 80 when the URL names none) whose tunnel then carries the opening handshake in plain text, with the fields of a wss:// one in the same order.

The proxy reads the tunnel’s first bytes. A TLS handshake is ended as above. GET is read as HTTP/1.1, and a WebSocket handshake in it takes steps 1 and 3 to 5 like a wss:// one, with Host checked against the CONNECT target; it goes upstream as http:// over a plain connection, so there is no ClientHello to impersonate.

Any other plain-text request gets an empty 502 (400 for half an upgrade) and the tunnel is closed, and bytes that start neither way close it unanswered: plain text is relayed only after an upgrade the site accepted, as for wss://.

A handshake outside a tunnel (an absolute-form http:// request, which Chrome does not send for ws://) is an empty 502. Through an egress, the plain handshake leaves the way any http:// request does, so an HTTP egress has to relay the upgrade itself; that is not tested.

The priming page#

Chrome gets the proxy’s token from ProxyAuth, which answers Fetch.authRequired for a tab’s requests. A WebSocket’s own CONNECT never gets there. Measured on Chrome 155 (headless): a wss:// WebSocket opened before any other request had gone through the proxy got the 407 on its CONNECT, Chrome did not retry it with the token, and the page saw an error; a Fetch.enable sent to the browser target saw nothing of it either. Credentials Chrome was given are kept in a cache that every tab of the browser context shares, and every later CONNECT carries them, a WebSocket’s included: one tab’s navigation, then a WebSocket as another tab’s first request, goes through.

So the proxy owns one page, mitm.prime_url(): https://sb-mitm.invalid/ (mimic::mitm::PRIME_HOST, under the reserved .invalid name) and a path of 128 random bits. After the token check, a GET of exactly that is answered by the proxy itself with an empty page that is never cached (mitm.stats().primed counts it). Any other request for that host, Chrome’s favicon.ico included, gets an empty 404. Nothing for that host goes upstream and the event sink hears nothing of it. The path holds no token; it is random so that a page that was not given it only ever sees the 404. configure_launch makes the priming page the start page, so Browser::launch loads it in the first tab before it returns, and a WebSocket may then be a new tab’s first request. A start page you set before configure_launch is kept (an http(s):// one takes the credentials just as well; an about: or data: one does not), and one set after it replaces the priming page.

A browser context keeps a cache of its own, and one created after the launch does not share the default context’s (measured). Before a WebSocket in a context behind the proxy, load the priming page in its first tab:

use seleniumbase_rs_mimic::MitmProxy;use seleniumbase_rs::sb_cdp::{Browser, ContextOptions};# async fn demo(browser: Browser, mitm: MitmProxy) -> Result<(), Box<dyn std::error::Error>> {let context = browser    .new_context_with(ContextOptions::new().proxy(mitm.browser_proxy()))    .await?;let page = context.new_page_at(mitm.prime_url()).await?;// WebSockets of this context now carry the proxy's credentials.# drop(page);# Ok(())# }

Changing the upstream, and stopping#

swap_upstream(egress) and swap_profile(profile) build the new upstream client first and then replace the old one. Requests under way finish on the old one, and the listener, its port and the browser’s open connections stay. shutdown(grace) stops accepting, lets requests finish for up to grace, then aborts what is left; dropping the last clone stops it too.

What is not intercepted#

What was run#