comparison
Hosted web scraping & search APIs, with open source alternatives
Every page below has to be fetched somehow — either you run the scraper, or you pay someone to. This is both sides of that decision: 12 hosted APIs with the pricing from their own pages, and 20 open source projects holding ★ 796K combined.
Hosted APIs
Closed services, metered by credit or request. Prices as published on each vendor's own pricing page. Each links to its open source alternatives.
| product | what it does | pricing | open source alternatives |
|---|---|---|---|
| Apify | Web scraping and automation platform with a hosted actor marketplace | Free plan available; usage metered per compute unit (about $0.2/unit on entry tiers, less on higher commitments) | 3 projects → |
| Bright Data | Proxy network and web data platform | Free trial; paid metered from roughly $0.2 to $1.3 per 1,000 requests depending on product | 3 projects → |
| Browserbase | Headless browser infrastructure for AI agents | Free plan; paid plans at $20/month and $99/month, plus custom | 3 projects → |
| Exa | Web search API built on embeddings for AI applications | Free tier: $20 in credits on sign-up plus $10/month, over $120/year | 1 project → |
| Fastcrawl | Scraping and crawling API for AI agents, over REST and MCP | Free tier 2,000 credits/month; paid from $5/month | 4 projects → |
| Firecrawl | Scrape, crawl and search the web for AI applications | Free tier 1,000 credits/month; paid plans from $19/month (billed annually) | 4 projects → |
| Oxylabs | Proxy network and web scraping APIs | Datacenter proxies from around $6/GB; scraping APIs from $49/month | 3 projects → |
| ScraperAPI | Scraping API handling proxies, browsers and retries | Paid plans from $49/month, scaling into the hundreds per month | 3 projects → |
| ScrapingBee | Web scraping API with proxy rotation and JavaScript rendering | Paid plans from $19/month, up to $599/month for the largest tier | 3 projects → |
| SerpApi | Search engine results API across Google and other engines | Free tier 100 searches/month; paid plans from $75/month | 1 project → |
| Tavily | Search API built for LLM applications and agents | Free tier 1,000 API credits/month; paid plans for higher monthly limits | 1 project → |
| Zyte | Web scraping API and data extraction platform | Free $5 trial credit; metered from about $0.06 per 1,000 successful responses | 3 projects → |
Open source projects
Self-hostable, ranked by GitHub stars. Each profile carries a written overview, licence and activity.
| project | stars | licence | written in |
|---|---|---|---|
| Browser Use Let AI agents control any browser, at scale | ★ 116K | MIT | Python |
| Crawl4AI LLM-ready web crawler built for AI data pipelines | ★ 84K | Apache-2.0 | Python |
| Scrapling 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDce | ★ 83K | BSD-3-Clause | Python |
| scrapy Scrapy, a fast high-level web crawling & scraping framework for Python. | ★ 64K | BSD-3-Clause | Python |
| EasySpider A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。 | ★ 45K | AGPL-3.0 | JavaScript |
| Lightpanda Fast, AI-native web browser built for automation and scraping | ★ 35K | AGPL-3.0 | Zig |
| changedetection.io Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock | ★ 34K | Apache-2.0 | Python |
| lux 👾 Fast and simple video download library and CLI tool written in Go | ★ 32K | MIT | Go |
| CloakBrowser Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed. | ★ 32K | MIT | Python |
| Scrapegraph-ai Python scraper based on AI | ★ 31K | MIT | Python |
| page-agent JavaScript in-page GUI agent. Control web interfaces with natural language. | ★ 29K | MIT | TypeScript |
| ArchiveBox 🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more... | ★ 29K | MIT | Python |
| crawlee Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or G | ★ 26K | Apache-2.0 | TypeScript |
| colly Elegant Scraper and Crawler Framework for Golang | ★ 26K | Apache-2.0 | Go |
| stagehand The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more. | ★ 25K | MIT | TypeScript |
| proxy_pool Python ProxyPool for web spider | ★ 24K | MIT | Python |
| Skyvern Automate any website workflow with AI and computer vision | ★ 23K | AGPL-3.0 | Python |
| Automa Build AI agents and RPA workflows without writing code | ★ 22K | NOASSERTION | Vue |
| Douyin_TikTok_Download_API 🚀 Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for posts, profiles, comments and pla | ★ 20K | Apache-2.0 | Python |
| katana A next-generation crawling and spidering framework. | ★ 18K | MIT | Go |
full list: all data extraction & web scraping projects → · every head-to-head comparison →
How to choose
Run it yourself when volume is the constraint rather than effort. A self-hosted crawler has no per-page price, and the projects above are mature — Crawl4AI and Firecrawl for LLM-ready markdown, Crawlee and Scrapy for general crawling, Playwright and Puppeteer when you need a real browser.
Pay for a hosted API when the operational parts are the problem: rotating proxies, rendering JavaScript, retrying failures and spreading requests without getting blocked. That is what the credit buys, and it is genuinely tedious to build and keep working.
Watch the metering, not the headline. Prices here are per credit or per request, and JavaScript rendering usually costs more than a static fetch. A free tier of 1,000 credits is not comparable to 2,000 of a different unit — check what one credit buys before comparing monthly figures.
Prices are taken from each vendor's own pricing page and recorded on the product's page here. Vendors change them; the linked page is the source of truth.
Frequently asked questions
Should I self-host a scraper or pay for a scraping API?
Self-host if you have the infrastructure and want no per-page cost — 20 open source projects here do this, led by Browser Use (116K stars). Pay for a hosted API if you would rather not run browsers, rotate proxies and handle retries yourself; all 12 hosted APIs below are metered by credit or request.
What is the cheapest hosted scraping API?
Of the hosted APIs listed here, entry pricing starts at $5/month (Fastcrawl), $19/month (Firecrawl, billed annually), $20/month (Browserbase) and $49/month (ScraperAPI, Oxylabs). Several offer a recurring free tier instead — 2,000 credits/month on Fastcrawl, 1,000 on Firecrawl and Tavily.
Which open source scraping projects are the most popular?
Ranked by GitHub stars: Browser Use (116K), Crawl4AI (84K), Scrapling (83K), scrapy (64K), EasySpider (45K). Combined, the projects listed here hold 796,242 stars.
Is Firecrawl open source?
Firecrawl is open source (AGPL-3.0) and also sells a hosted cloud. Self-hosting the core is possible; the managed tier is what the $19/month entry price buys. It appears on both sides of this page for that reason.