Obscura CLI Guide — Seven Output Formats and Concurrent Scraping

One Command, One Browser

The Obscura CLI is the most direct way to use the project. Download a single binary, no Chrome, no Node.js, no dependencies — start scraping immediately.

The CLI entry point is obscura, with four subcommands:

• obscura fetch <URL> — fetch a single page and output data
• obscura serve — start a CDP WebSocket server for Puppeteer/Playwright
• obscura scrape <URL...> — fetch multiple pages concurrently
• obscura mcp — start an MCP server for AI agents

This article focuses on fetch and scrape.

Installation

Download the binary for your platform (the URL below is a placeholder; replace it with the actual release page):

# Linux x86_64
curl -LO https://example.16yun.cn/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
tar xzf obscura-x86_64-linux.tar.gz
 
# macOS Apple Silicon
curl -LO https://example.16yun.cn/obscura/releases/latest/download/obscura-aarch64-macos.tar.gz
tar xzf obscura-aarch64-macos.tar.gz

Verify:

./obscura --version
./obscura fetch https://example.com --eval "document.title" --quiet

obscura fetch: Seven Output Formats

fetch is the primary subcommand. The --dump flag controls the output format.

--dump html

Default format. Returns the full HTML string after JavaScript execution.

obscura fetch https://httpbin.org/anything --dump html

--dump text

Plain text output. Strips all markup and scripts.

obscura fetch https://httpbin.org/anything --dump text

--dump markdown

Converts the rendered page to Markdown. Preserves headings, lists, links, code blocks, images, and tables.

obscura fetch https://example.16yun.cn/help/quickstart/ --dump markdown > page.md

This is the recommended format for LLM / RAG pipelines.

--dump links

Extracts all <a href> links, one per line.

obscura fetch https://example.com --dump links

Useful for SEO analysis, link checking, and sitemap construction.

--dump assets

Outputs all sub-resource URLs as NDJSON, including:

• Stylesheets, scripts, images
• Fonts, iframes, embeds
• JavaScript fetch/XHR requests

obscura fetch https://example.com --dump assets

--dump original

Returns the raw HTTP response body. Binary-safe.

obscura fetch https://httpbin.org/image/png --dump original > photo.jpg

--dump cookies

Outputs all cookies in the browser jar as a JSON array, including HttpOnly cookies.

obscura fetch https://example.com --dump cookies

Useful for extracting session tokens set by anti-bot challenges.

--eval: Execute JavaScript

Run arbitrary JavaScript after page load:

obscura fetch https://httpbin.org/anything --eval "JSON.stringify(Array.from(document.querySelectorAll('.titleline > a')).map(a => ({title: a.textContent, url: a.href})))"

A multi-statement --eval starting with const returns null (V8 const completion value is empty). Wrap in an IIFE:

obscura fetch https://example.com --eval "(function(){ return document.title; })()"

Wait Strategies

SPA pages need to wait for JavaScript execution. --wait-until controls the condition:

# Wait for load event (default)
obscura fetch https://example.16yun.cn --wait-until load
 
# Wait for DOMContentLoaded
obscura fetch https://example.16yun.cn --wait-until domcontentloaded
 
# Wait for network idle (no requests for 500ms)
obscura fetch https://example.16yun.cn --wait-until networkidle0
 
# Wait for network basic idle (allows up to 2 connections)
obscura fetch https://example.16yun.cn --wait-until networkidle2

--wait adds extra settle time after the condition is met:

obscura fetch https://example.16yun.cn --wait 10

--selector: Narrow to a Region

Restrict output to elements matching a CSS selector:

obscura fetch https://example.com --selector "main" --dump markdown
obscura fetch https://example.com --selector "article.post" --dump text

--timeout and Process-Level Hard Deadline

--timeout controls the navigation timeout (default 30 seconds):

obscura fetch https://example.16yun.cn --timeout 60

A process-level daemon thread force-exits if timeout + wait + 10 seconds elapse. This is the absolute backstop — the V8 watchdog cannot terminate synchronous Rust ops, so the hard deadline ensures one bad page never wedges a worker.

obscura scrape: Concurrent Scraping

scrape fans out across worker processes:

obscura scrape url1 url2 url3 --concurrency 25 --format json

Flags:

• --concurrency: Worker count (default 10)
• --format: json or text
• --timeout: Per-worker timeout (default 60s)
• --eval: JS expression per page
• --quiet: Suppress progress output

Read URLs from stdin:

cat urls.txt | obscura scrape --concurrency 20 --format json -

Requires obscura-worker next to obscura in the same directory.

Output and Pipes

Write output to a file:

obscura fetch https://example.com --dump markdown --output page.md
obscura fetch https://example.com --eval "document.title" -o title.txt

--quiet strips logs for script-friendly output:

obscura fetch https://example.com --dump text --quiet | wc -w

Unix pipelines:

obscura fetch https://example.com --dump links --quiet | grep 'blog' | head -10

Global Flags

These apply to all subcommands:

obscura --proxy socks5://127.0.0.1:1080 --stealth fetch https://example.com --dump markdown

End-to-End Example

# Set proxy
export OBSCURA_PROXY=http://user:pass@proxy.16yun.cn:8888
 
# Fetch SPA with networkidle0, extract main content as Markdown
obscura fetch https://example.16yun.cn \
  --wait-until networkidle0 \
  --selector "main" \
  --dump markdown \
  --output article.md \
  --stealth
 
# Extract page title
obscura fetch https://example.16yun.cn \
  --eval "document.title" \
  --quiet

Environment Variables

export OBSCURA_NAV_TIMEOUT_MS=60000
export OBSCURA_FETCH_TIMEOUT_MS=15000
obscura fetch https://example.16yun.cn --dump text

Summary

obscura fetch and obscura scrape cover everything from single-page fetching to concurrent batch scraping. Seven output formats serve different downstream needs, --eval provides unlimited flexibility, and the wait strategy / timeout system ensures reliable handling of SPAs and slow pages.