crw (fastCRW) is a Rust-based web scraping, crawling, and search engine with an MCP server that turns URLs into clean markdown or structured JSON, aimed at AI agents and the developers who wire them into coding tools.

What it is

crw is a single-engine tool for search, scrape, map, crawl, and extract, distributed both as a small Rust binary that runs locally and as a managed cloud API. It lives in the AI-agent data-extraction ecosystem: the output formats are markdown, HTML, links, screenshots, and schema JSON, which is the shape language models and agent pipelines consume. The project ships under AGPL-3.0, with documentation at docs.fastcrw.com and a hosted service at fastcrw.com.

The concrete problem it solves is replacing a paid hosted scraping and search API with something that can run on the user's own machine or infrastructure. crw exposes a drop-in Firecrawl-compatible REST surface at /scrape, /crawl, and /search, so code written against Firecrawl's endpoints can be pointed at crw instead. The same server also registers itself as an MCP server, which removes the glue work of hand-wiring a scraper into an agent client. Self-hosters keep their own data and choose their own renderers, search backend, authentication, and proxies.

Key capabilities

• Five documented operations: scrape one URL, run a bounded site crawl, map URLs without scraping each page, search the web and optionally scrape results, and extract structured fields from one or many URLs.
• CLI entry points, including crw https://example.com for an immediate scrape and crw search "rust async runtime" for search once crw setup has run.
• Firecrawl-compatible REST API at /scrape, /crawl, and /search, plus a capabilities endpoint at /v1/capabilities.
• MCP server installation through npx -y crw-mcp@latest install, which installs the CRW skill and MCP server into detected AI tools; crw setup can perform the same step.
• Client SDKs for pip install crw (the CrwClient class) and npm install crw-sdk, both supporting a formats option such as ["markdown"].
• Output formats of markdown, HTML, links, screenshots, and schema JSON from a single scrape call.
• Performance claims published against two benchmarks: 2.3x faster than Tavily and 1.5x faster than Firecrawl on a 1,000-URL benchmark, and 90.0% answer accuracy on the 600-question AA-Omniscience set.

Who uses it and how

• Developers inside AI coding agents. Installation detects Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf when they are already configured, registers the MCP server, and leaves the API key in ~/.config/crw/config.toml rather than copying it into each tool.
• Teams that need data control, private networks, or custom infrastructure, who run the binary locally and select their own renderers, search, auth, proxies, and capacity.
• Python and Node.js application developers who consume scraping through the crw and crw-sdk packages instead of driving the CLI.
• Teams with no infrastructure to operate, who use the managed API for proxy handling and JavaScript rendering, starting from 1,000 free credits with no credit card.
• Evaluation and benchmarking users comparing search-answer accuracy across the 14-product field documented in the project's benchmark material.

Getting started

Install with the one-command script curl -fsSL https://fastcrw.com/install | sh, which runs locally and free with no account; add CRW_API_KEY=crw_live_... to the same command to connect a cloud key and register the MCP server. SDK users instead run pip install crw or npm install crw-sdk.

How it compares

The project positions itself as an alternative to Firecrawl and Tavily, and it is licence-transparent in a way that matters for self-hosting: the source is AGPL-3.0, so a team can deploy the whole engine itself rather than relying on a vendor endpoint. Compatibility is at the API level, since crw serves the same /scrape, /crawl, and /search paths, which makes migration a matter of changing the base URL rather than rewriting client code. crw also offers a managed cloud for teams that would rather not run anything, so the same project spans the free self-hosted case and the paid hosted case.

When to use it — and when not to

A self-hoster must operate the pieces the local deployment leaves to the user: renderers, search, authentication, proxies, and capacity, and the documentation warns that capabilities and response shapes can differ between the managed and local deployments. Anyone on Windows should check compatibility first, since the install is documented for macOS and Linux on Intel and ARM only. The reported memory footprint is inconsistent between the repository description, which cites 6 MB, and the README benchmark, which reports roughly 14 MB idle, so capacity planning should be based on measurement rather than either figure.

curl -fsSL https://fastcrw.com/install | sh

curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh

crw search "rust tutorials"

crw https://example.com            # scrape, works right after install
crw search "rust async runtime"    # search, after `crw setup`

export CRW_API_KEY="crw_live_..."
pip install crw

from crw import CrwClient

client = CrwClient()
page = client.scrape("https://example.com", formats=["markdown"])

print(page["markdown"])

npm install crw-sdk

import { CrwClient } from "crw-sdk";

const client = new CrwClient();

const page = await client.scrape("https://example.com", {
  formats: ["markdown"],
});

console.log(page.markdown);

npx -y crw-mcp@latest install

git clone https://github.com/us/crw
cd crw
make check-fast