Spider is a concurrency-first web crawling and scraping engine written in Rust, licensed under MIT, and aimed at developers and data teams that need to collect web data with low latency, from a single script up to a distributed fleet.

What it is

Spider is a Rust library that crawls and scrapes websites, and it lives in the Rust crate ecosystem as the spider crate on crates.io. From that core, the project ships the same engine to other runtimes: a command-line tool as spider_cli, a Node.js package as @spider-rs/spider-rs on npm, a Python package as spider_rs on PyPI, and an MCP server as spider_mcp for tools such as Claude and Cursor. The engine runs HTTP-first and launches headless Chrome only for pages that need JavaScript, and both the HTTP and the Chrome paths stream pages back as they are fetched rather than batching results at the end. The same engine also runs Spider Cloud, the project's managed service.

The concrete problem it solves is the infrastructure around crawling rather than the crawling code itself. The README states plainly that the hardest part of crawling at scale is proxies, headless browsers, and constant anti-bot churn. Spider builds proxies, retries, rate limiting, and stealth into the engine, so a crawler does not have to assemble a separate HTTP client, a headless-browser driver, and a proxy and retry layer by hand. A single Website object carries the limits, depth, delay, robots handling, user agent, and stealth settings, and the same API and concurrency model drive one async task or a distributed worker fleet, so moving from a local script to managed infrastructure is a configuration change rather than a rewrite.

Key capabilities

• Streaming page delivery through a channel: Website::new, then subscribe(16) returns a receiver that yields pages as they arrive, with crawl() and unsubscribe() closing the run.
• Selective JavaScript rendering: enabling the chrome feature and calling crawl_smart() makes Spider try HTTP first and start Chrome only on pages that need it.
• Configuration surface with sensible defaults: with_limit(50) for concurrent requests, with_depth(10) for link depth, with_delay(500) in milliseconds, with_respect_robots_txt(true), with_subdomains(true), with_user_agent(Some("MyBot/1.0")), and with_stealth(true).
• Managed mode through the spider_cloud feature: SpiderCloudConfig::new("sk-...") with SpiderCloudMode::Smart routes through proxies first and escalates to the unblocker only on pages that fight back.
• Built-in proxies, retries, rate limiting, and stealth across both the HTTP and Chrome paths.
• Page export as Markdown, JSON, or WARC.
• Over 50 runnable examples in the examples directory, plus an MCP server for agent tooling.

Who uses it and how

• Teams feeding the open web into vector stores for LLM and RAG pipelines, using the streaming API so pages can be embedded as they are fetched.
• SEO and price-monitoring workflows that re-crawl sites on a schedule and diff the exported Markdown, JSON, or WARC output.
• AI browsing agents that need headless Chrome driven by the same Rust engine.
• Polyglot teams that adopt the crawler without adopting Rust for the whole stack: cargo add spider for Rust, npm i @spider-rs/spider-rs for Node.js, pip install spider_rs for Python.
• Agent developers who install spider_mcp and expose crawling to Claude or Cursor as an MCP tool.

Getting started

Add the crate with cargo add spider for the library, or install the CLI with cargo install spider_cli; the Node.js, Python, and MCP entry points are npm i @spider-rs/spider-rs, pip install spider_rs, and cargo install spider_mcp. For managed crawling, Spider Cloud issues a free API key without a card and is enabled in Rust by depending on spider version 2 with the spider_cloud feature.

How it compares

The supplied facts name no competing crawler or scraping products, so Spider stands alone in this registry rather than being contrasted against named alternatives. The only adjacent offering the facts describe is Spider Cloud, which runs the same engine as a managed service with proxies, headless browsers, and anti-bot handling included.

When to use it — and when not to

Choose Spider when the crawl logic is the part the team wants to own and the result should stay in its own infrastructure under the MIT licence. Self-hosters must still operate what the engine depends on at scale, above all headless Chrome for JavaScript-heavy sites, and they carry the proxy pool, retry, rate-limit, and anti-bot work themselves, which the README concedes is the hard part. Teams that do not want to run that infrastructure should use Spider Cloud behind the same API instead, and anyone needing a fully managed, zero-operations crawler with no self-hosted footprint should not pick the local library path.

[dependencies]
spider = { version = "2", features = ["spider_cloud"] }

use spider::configuration::{SpiderCloudConfig, SpiderCloudMode};
use spider::website::Website;

let cloud = SpiderCloudConfig::new("sk-...")
    .with_mode(SpiderCloudMode::Smart); // proxy by default, auto-unblock when blocked

let mut website = Website::new("https://example.com")
    .with_spider_cloud_config(cloud)
    .build()?;

[dependencies]
spider = "2"

use spider::{tokio, website::Website};

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{}  {}", page.status_code, page.get_url());
        }
    });

    website.crawl().await;
    website.unsubscribe();
}

let mut website = Website::new("https://example.com")
    .with_limit(50)                    // concurrent requests
    .with_depth(10)                    // how deep to follow links
    .with_delay(500)                   // pause between requests (ms)
    .with_respect_robots_txt(true)
    .with_subdomains(true)
    .with_user_agent(Some("MyBot/1.0"))
    .with_stealth(true)
    .build()
    .unwrap();

cargo test -p spider                  # unit tests
RUN_LIVE_TESTS=1 cargo test           # live network tests