firecrawl is a free, open source miscellaneous project written in TypeScript and released under AGPL-3.0. It has 181,627 GitHub stars, 9,834 forks and 621 open issues, and was last pushed 3 hours ago. On this registry it ranks #1 of 25 tracked projects in Miscellaneous, with 5 head-to-head comparisons available. It gained 2,106 stars over the last 6 tracked days.

What is firecrawl?

What it is

Firecrawl is an open-source web context API that enables AI agents to search, scrape, and interact with websites at scale. It lives in the AI and data extraction ecosystem, providing structured, LLM-ready outputs from unstructured web content. Built in TypeScript and licensed under AGPL-3.0, it supports both self-hosting and a hosted service via firecrawl.dev.

It solves the problem of unreliable, slow, or configuration-heavy web scraping for AI workflows. Traditional scrapers often fail on JavaScript-heavy pages, require proxy rotation, and produce noisy HTML. Firecrawl handles JS rendering, proxy management, rate limiting, and content cleaning automatically, delivering clean Markdown or structured JSON suitable for direct use in LLM pipelines.

Key capabilities

  • Search the web and retrieve full-page markdown content from search results
  • Scrape any URL to Markdown, HTML, screenshots, or structured JSON
  • Interact with pages programmatically: click, scroll, type, wait, and extract after actions
  • Crawl entire sites with a single request, discovering and scraping all URLs
  • Batch scrape thousands of URLs asynchronously
  • Parse web-hosted PDFs, DOCX, and other media files
  • Map a domain to list all discoverable URLs instantly

Who uses it and how

AI agent developers integrate Firecrawl to gather real-time web context before tool execution or planning steps. Data engineers use it to populate vector databases or knowledge graphs with clean, structured outputs from public sites. Researchers and product teams run batch crawls to collect competitive intelligence or market data at scale.

Getting started

Deploy via Docker (firecrawl/firecrawl) or install the CLI (npm install -g @mendable/firecrawl). Sign up at firecrawl.dev for a hosted API key and use the playground to test endpoints. SDKs exist for Python and Node.js; cURL and CLI commands are documented.

When to use it — and when not to

Use Firecrawl when you need reliable, JS-capable scraping with minimal setup for AI agents or batch data extraction. Avoid it if you require fine-grained control over request headers, custom authentication flows, or compliance with strict legal restrictions on automated access. Self-hosting requires PostgreSQL for metadata storage and S3-compatible storage for screenshots; SMTP is optional for notifications. The AGPL-3.0 license mandates derivative works to be open-sourced, which may not suit proprietary systems.

project readme (upstream, from github) — read inline

License Downloads GitHub Contributors Visit firecrawl.dev

Follow on X Follow on LinkedIn Join our Discord


🔥 Firecrawl

The API to search, scrape, and interact with the web at scale. 🔥 The web context API to find sources, extract content, and turn it into clean Markdown or structured data your agents can ship with. Open source and available as a hosted service.

Pst. Hey, you, join our stargazers :)

GitHub stars

Why Firecrawl?

  • Industry-leading reliability: Covers 96% of the web, including JS-heavy pages — no proxy headaches, just clean data (see benchmarks)
  • Blazingly fast: P95 latency of 3.4s across millions of pages, built for real-time agents and dynamic apps
  • LLM-ready output: Clean markdown, structured JSON, screenshots, and more — spend fewer tokens, build better AI apps
  • We handle the hard stuff: Rotating proxies, orchestration, rate limits, JS-blocked content, and more — zero configuration
  • Agent ready: Connect Firecrawl to any AI agent or MCP client with a single command
  • Media parsing: Parse and extract content from web-hosted PDFs, DOCX, and more
  • Actions: Click, scroll, write, wait, and press before extracting content
  • Open source: Developed transparently and collaboratively — join our community

Feature Overview

Core Endpoints

Feature Description
Search Search the web and get full page content from results
Scrape Convert any URL to markdown, HTML, screenshots, or structured JSON
Interact Scrape a page, then interact with it using AI prompts or code

More

Feature Description
Agent Automated data gathering, just describe what you need
Crawl Scrape all URLs of a website with a single request
Map Discover all URLs on a website instantly
Batch Scrape Scrape thousands of URLs asynchronously

Quick Start

Sign up at firecrawl.dev to get your API key. Try the playground to test it out.

Search

Search the web and get full content from results.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

search_result = app.search("firecrawl", limit=5)
Node.js / cURL / CLI

Node.js

import { Firecrawl } from 'firecrawl';

const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});

app.search("firecrawl", { limit: 5 })

cURL

curl -X POST 'https://api.firecrawl.dev/v2/search' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
  "query": "firecrawl",
  "limit": 5
}'

CLI

firecrawl search "firecrawl" --limit 5

Output:

[
  {
    "url": "https://firecrawl.dev",
    "title": "Firecrawl",
    "markdown": "Turn websites into..."
  },
  {
    "url": "https://docs.firecrawl.dev",
    "title": "Firecrawl Docs",
    "markdown": "# Getting Started..."
  }
]

Scrape

Get LLM-ready data from any website — markdown, JSON, screenshots, and more.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

result = app.scrape('firecrawl.dev')
Node.js / cURL / CLI

Node.js

import { Firecrawl } from 'firecrawl';

const app = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" });

app.scrape('firecrawl.dev')

cURL

curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
  "url": "firecrawl.dev"
}'

CLI

firecrawl scrape https://firecrawl.dev
firecrawl https://firecrawl.dev --only-main-content

Output:

# Firecrawl

Firecrawl helps AI agents search, scrape, and interact with the web.

## Features
- Search: Find information across the web
- Scrape: Clean data from any page
- Interact: Click, navigate, and operate pages
- Agent: Autonomous data gathering

Interact

Scrape a page, then interact with it using AI prompts or code.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

result = app.scrape("https://amazon.com")
scrape_id = result.metadata.scrape_id

app.interact(scrape_id, prompt="Search for 'mechanical keyboard'")
app.interact(scrape_id, prompt="Click the first result")
Node.js / cURL / CLI

Node.js

import { Firecrawl } from 'firecrawl';

const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});

const result = await app.scrape("https://amazon.com");

await app.interact(result.metadata.scrapeId, {
  prompt: "Search for 'mechanical keyboard'"
});
await app.interact(result.metadata.scrapeId, {
  prompt: "Click the first result"
});

cURL

# 1. Scrape the page
curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url": "https://amazon.com"}'

# 2. Interact with the page (use scrapeId from step 1)
curl -X POST 'https://api.firecrawl.dev/v2/scrape/SCRAPE_ID/interact' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"prompt": "Search for mechanical keyboard"}'

CLI

firecrawl scrape https://amazon.com
firecrawl interact exec --prompt "Search for 'mechanical keyboard'"
firecrawl interact exec --prompt "Click the first result"

Output:

{
  "success": true,
  "output": "Keyboard available at $100",
  "liveViewUrl": "https://liveview.firecrawl.dev/..."
}

Power Your Agent

Connect Firecrawl to any AI agent or MCP client in minutes.

Skill

Give your agent easy access to real-time web data with one command.

npx -y firecrawl-cli@latest init --all --browser

Restart your agent after installing. Works with Claude Code, Antigravity, OpenCode, and more.

MCP

Connect any MCP-compatible client to the web in seconds.

{
  "mcpServers": {
    "firecrawl-mcp": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "fc-YOUR_API_KEY"
      }
    }
  }
}

Agent Onboarding

Are you an AI agent? Fetch this skill to sign up your user, get an API key, and start building with Firecrawl.

curl -s https://firecrawl.dev/agent-onboarding/SKILL.md

See the Skill + CLI documentation for all available commands. For MCP, see firecrawl-mcp-server.


More Endpoints

Agent

The easiest way to get data from the web. Describe what you need, and our AI agent searches, navigates, and retrieves it. No URLs required.

Agent is the evolution of our /extract endpoint: faster, more reliable, and doesn't require you to know the URLs upfront.

curl -X POST 'https://api.firecrawl.dev/v2/agent' \
  -H 'Authorization: Bearer fc-YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "pr

readme truncated — read the full docs on github

Frequently asked questions

Is firecrawl free to use?

firecrawl is open source under the AGPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does firecrawl do?

firecrawl is an open source project.

What is firecrawl written in?

firecrawl is primarily written in TypeScript. Its source is publicly available at https://github.com/mendableai/firecrawl, and it has 181,627 GitHub stars.