Obscura Architecture: 8 Crates, V8 Isolate, and CDP Dispatch

From Command to Bytes

When Puppeteer calls page.goto('https://example.com'), what happens inside Obscura? This article traces that path through the 8 crates and explains why the architecture is split this way. The point of understanding the architecture is not to read source — it is to know which layer to look at when you are debugging, tuning, or extending.

The 8 Crates

obscura-cli       CLI entry: fetch / serve / scrape / mcp
obscura-cdp       CDP WebSocket: routing, domain handlers
obscura-browser   Page type: navigation, lifecycle, context
obscura-js        V8 runtime: deno_core, bootstrap.js, Rust ops
obscura-dom       DOM tree: html5ever parsing, selectors, serialization
obscura-net       HTTP: reqwest, wreq stealth client, cookie jar, tracker blocklist
obscura-mcp       MCP protocol server: 30+ tools
obscura           Embeddable library API: Browser / Page / Element / CookieStore

The cross-crate rule: upper layers call lower layers, never sideways. This rule tells you where to look during troubleshooting — a network-layer symptom will not disappear because of a DOM-layer change, and vice versa.

Request Flow

A Page.navigate from a CDP client:

CDP Client (Puppeteer)
       │ WebSocket frame
       ▼
obscura-cdp/server.rs          accept, route by sessionId
       │
       ▼
obscura-cdp/dispatch.rs        method router, acquires v8_lock
       │
       ▼
obscura-cdp/domains/page.rs    Page.navigate handler
       │
       ▼
obscura-browser/page.rs        navigate_with_wait
       │
       ├──► obscura-net/client.rs       HTTP fetch
       │
       ├──► obscura-dom/tree.rs         parse HTML into DOM tree
       │
       └──► obscura-js/runtime.rs       run inline scripts
                 │
                 └──► bootstrap.js + ops.rs   DOM bindings

The dispatcher emits CDP events (Network.requestWillBeSent, Page.frameNavigated, Page.lifecycleEvent) back through the same WebSocket. This event chain is what Puppeteer and Playwright use to decide "the page has loaded" — the client only resolves goto after it receives the matching lifecycleEvent.

Single V8 Isolate

All pages share one V8 Isolate. This is Obscura's most important architectural decision and the root cause of its low memory footprint — a Chrome process allocates a separate renderer process and Isolate per tab, while Obscura serves all pages from one Isolate.

let _guard = obscura_js::v8_lock::global().lock().await;
page.evaluate(expr).await;

The V8 Isolate is single-threaded by design, so any operation that runs JavaScript must acquire this global lock first:

Runtime.evaluate, callFunctionOn
Page.navigate (needs to run page scripts)
DOM operations (node insertion, attribute queries)

The dispatcher routes long-running operations (navigation, eval) through process_with_interception, which spawns them onto a tokio LocalSet so the dispatcher keeps handling other CDP messages. This is why Target.createTarget from many concurrent clients returns immediately: each newPage resolves at once while the actual navigation runs in a background task.

This design has direct implications for how you use it:

Bound your concurrent page count. newPage returns immediately, but all pages' JS execution serializes through the same lock. Running 50 pages with heavy JS is effectively 50 tasks queuing for one lock.
CPU-bound JS drags other pages. A page running a tight loop (which the watchdog will eventually terminate) blocks every other page's JS execution until it is killed.
Use multi-process for real parallelism. obscura serve --workers N spawns multiple processes, each with its own Isolate — that is the correct way to scale horizontally.

DOM Tree

Obscura uses html5ever for HTML parsing with a custom tree_sink, and Servo's selectors crate as the selector engine.

// obscura-dom/src/tree.rs
struct Node {
    id: NodeId,
    parent: Option<NodeId>,
    children: Vec<NodeId>,
    node_type: NodeType,  // Element / Text / Comment / Document
}
 
struct Tree {
    nodes: Vec<Node>,
    root: NodeId,
}

A few design properties that affect usage:

Cycle guards. append_child and insert_before reject inserting an ancestor as a child. These tree-operation guards ensure the DOM tree never contains a cycle, so traversals like descendants() always finish in a finite number of steps.
descendants() cap. As an extra safeguard, tree traversal carries a hard-coded length cap so that even an anomalous structure cannot recurse without bound.
Selector engine. Built on the selectors crate (Servo's CSS selector implementation) with standard CSS selector syntax, so the selectors you use via CDP querySelector or the CLI --selector behave the same as in a real browser.

CDP Dispatch

obscura-cdp is the WebSocket server. Its core pieces are server.rs (accept connections), dispatch.rs (route by sessionId), and domains/*.rs (one handler per CDP domain).

Session model. Each CDP client connection attaches to one or more targets. Session IDs take the form "{targetId}-session", and the dispatcher routes by the sessionId in the incoming frame to the right Page.

Target creation. Target.createTarget creates a new Page. Closing the WebSocket detaches all sessions but leaves the pages running — unlike Chrome, a disconnect does not destroy page state.

Long operations. process_with_interception extracts long-running operations (navigation, JS eval) and spawns them into independent tasks, freeing the dispatcher for other messages. This is why a single slow page's navigation does not stall other sessions' commands.

Lifecycle State Machine

Managed by obscura-browser/lifecycle.rs:

init → commit → domcontentloaded → load → networkidle2 → networkidle0

init: navigation initiated
commit: first response byte received
domcontentloaded: DOM parsed
load: all synchronous resources loaded
networkidle2: ≤ 2 active connections for 500ms
networkidle0: no active connections for 500ms

Page.navigate's waitUntil parameter maps onto these events. Puppeteer's goto waits client-side for the matching Page.lifecycleEvent.

Understanding this chain helps debug goto hangs: if goto is stuck, the root cause is that some lifecycleEvent never arrives. Reading the event sequence in obscura serve --verbose logs quickly tells you whether the problem is the network layer (load never comes) or the page layer (networkidle keeps getting reset by long polling).

Storage Persistence

--storage-dir enables persistent storage:

<storage-dir>/
├── cookies.json
└── localStorage/
    ├── <origin-1>.json
    └── <origin-2>.json

Read on startup, written on every navigation and on graceful shutdown. For multi-identity scenarios, run several obscura serve processes, each with a different --storage-dir and port, fully isolated from each other.

Stealth Integration

--stealth is a global CLI flag (not a subcommand flag) that applies to all subcommands:

obscura fetch https://example.16yun.cn --stealth
obscura serve --stealth
obscura scrape url1 url2 --stealth
obscura mcp --stealth

In scrape, it is forwarded to workers via OBSCURA_STEALTH.

Stealth swaps the default reqwest client in obscura-net for the wreq client, and applies JS API masking in bootstrap.js under obscura-js. The toggle only touches two crates and is transparent to the others — a benefit of the layered design.

Why 8 Crates

The layering is not cosmetic; it carries real engineering value:

Single responsibility. Each crate covers one capability domain, so newcomers locate code quickly. Changing DOM does not require reading the network layer; changing CDP dispatch does not require touching V8.
Compile caching. Modifying one crate does not rebuild V8. V8's source compile is the most expensive part (~5 minutes cold), and layering keeps incremental builds of everyday iteration down to seconds.
Test boundaries. Each crate can be tested independently — DOM tests need no networking, CDP tests need no V8. Unit tests stay fast and pinpoint.
Optional dependencies. Stealth lives only in obscura-net and does not affect the core. Users who do not need anti-detection build with the default rustls and pull in neither BoringSSL nor cmake.

This layering also defines the seams for extending Obscura: add a CDP method under obscura-cdp/domains/; add a Web API with a JS shim in bootstrap.js plus a Rust op; change network behavior in obscura-net. Every kind of extension has a clear home layer and needs no cross-layer surgery.

Summary

Obscura's architecture is a series of performance-constrained decisions: a single V8 Isolate avoids thread-synchronization overhead, Rust directly calls V8's C API without a bridge layer, and the DOM tree lives in Rust without serialization overhead. The 8-crate layering gives every change, test, and extension a clear boundary. Understanding the architecture is about knowing which layer to look at when debugging, which dial to turn when tuning, and where to start when extending.

Obscura Architecture — 8 Crates, Single V8 Isolate, and CDP Dispatch