Open Source Data Extraction Web Scraping Tools

Browser Use ★ 120K

Let AI agents control any browser, at scale

Scrapling ★ 92K

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDce

Crawl4AI ★ 87K

LLM-ready web crawler built for AI data pipelines

scrapy ★ 65K

Scrapy, a fast high-level web crawling & scraping framework for Python.

EasySpider ★ 45K

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

Lightpanda ★ 40K

Fast, AI-native web browser built for automation and scraping

changedetection.io ★ 35K

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock

CloakBrowser ★ 33K

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.

Scrapegraph-ai ★ 32K

Python scraper based on AI

dots ★ 32K

Open-source dots for the web: an AI agent with its own browser, one that does not get blocked.

lux ★ 32K

👾 Fast and simple video download library and CLI tool written in Go

page-agent ★ 30K

JavaScript in-page GUI agent. Control web interfaces with natural language.

ArchiveBox ★ 29K

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

crawlee ★ 26K

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or G

stagehand ★ 26K

The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.

colly ★ 26K

Elegant Scraper and Crawler Framework for Golang

proxy_pool ★ 24K

Python ProxyPool for web spider

Skyvern ★ 23K

Automate any website workflow with AI and computer vision

moli ★ 22K

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

Automa ★ 22K

Build AI agents and RPA workflows without writing code

Douyin_TikTok_Download_API ★ 21K

🚀 Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for posts, profiles, comments and pla

Maxun ★ 18K

No-code web scraping, crawling, and extraction platform

katana ★ 18K

A next-generation crawling and spidering framework.

SeleniumBase ★ 13K

APIs for browser automation, testing, and bypassing bot-detection. Includes CDP Mode: A stealthy configuration for chromium that passes every bot detection test

crawlab ★ 12K

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

camofox-browser ★ 12K

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

webmagic ★ 12K

A scalable web crawler framework for Java.

jsoup ★ 11K

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

nopecha-extension ★ 11K

Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.

pinchtab ★ 11K

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

OpenBot ★ 9.6K

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Brin

crawlee-python ★ 9.6K

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, P

Steel ★ 7.9K

Cloud browser sessions for AI agents and web automation

JMComic-Crawler-Python ★ 7.9K

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

firecrawl-mcp-server ★ 7.7K

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

mcp ★ 7.3K

Browser MCP is a Model Context Provider (MCP) server that allows AI applications to control your browser

google-maps-scraper ★ 7.2K

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,emai

rod ★ 7.1K

A Chrome DevTools Protocol driver for web automation and scraping.

pydoll ★ 7.1K

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

trafilatura ★ 7.1K

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

curl_cffi ★ 6.8K

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

node-crawler ★ 6.8K

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

open-dots ★ 6.5K

Open-source alternative to OpenAI Dots: self-hosted AI chat, tools, approvals, connectors, and computer tasks.

skills ★ 6.3K

Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution, indepen

ferret ★ 6.0K

Declarative data automation language and Go runtime for structured extraction workflows.

patchright ★ 5.0K

Undetected version of the Playwright testing and automation library.

Noria ★ 4.5K

An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices

design-extract ★ 4.5K

Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-plat

workflow-use ★ 4.2K

⚙️ Create and run workflows (RPA 2.0)

DotnetSpider ★ 4.1K

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

scrape-it ★ 4.1K

🔮 A Node.js scraper for humans.

puppeteer-sharp ★ 3.9K

Headless Chrome .NET API

anakin ★ 3.9K

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make

cariddi ★ 3.8K

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

toapi ★ 3.6K

Every web site provides APIs.

playwright-go ★ 3.5K

Playwright for Go a browser automation library to control Chromium, Firefox and WebKit with a single API.

atomic-agent ★ 3.5K

Atomic Agent is a local-first AI agent. Runs open-weight models on your own machine via llama.cpp.

stealth-browser-mcp ★ 3.3K

The only browser automation that bypasses anti-bot systems. AI writes network hooks, clones UIs pixel-perfect via simple chat.

VirtualBrowser ★ 3.2K

Free anti fingerprint browser, 指纹浏览器, 隐私浏览器, 防识别浏览器, 反识别浏览器, 防关联浏览器, 免费的web3空投专用指纹浏览器 https://virtualbrowser.cc/?src=github

browser-control ★ 3.2K

A tiny, fast Rust CLI that drives a real browser over the Chrome DevTools Protocol — built for coding agents.

FinalRecon ★ 3.0K

All In One Web Recon

js-reverse-mcp ★ 3.0K

AI Agent-first JS 逆向 MCP Server：有头 Chrome 调试、断点、网络/WebSocket 分析、Patchright 反检测，可选 CloakBrowser。

google-play-scraper ★ 3.0K

Node.js scraper to get data from Google Play

twscrape ★ 2.9K

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

spider ★ 2.8K

Foundational low latency web data collecting in Rust

geziyor ★ 2.8K

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

brightdata-mcp ★ 2.7K

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

HolyClaude ★ 2.6K

AI coding workstation: Claude Code + web UI + 8 AI CLIs + headless browser + 50+ tools

invisible_playwright ★ 2.5K

Free antidetect browser stealth for Playwright: undetected headless Firefox fingerprint. Python scraping, recaptcha and bot detection bypass. Open source

grab ★ 2.5K

Web Scraping Framework

webclaw ★ 2.4K

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

reverse-api-engineer ★ 2.4K

The agent that turns websites into APIs!

lcu ★ 2.4K

Codex computer use, decoupled from the app, for usage inside any harness.

abot ★ 2.3K

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

recipe-scrapers ★ 2.3K

Python package for scraping recipes data

google-maps-scraper-kit ★ 2.1K

Free Google Maps scraper for lead generation. Exports phones, emails, websites and socials to CSV. Runs locally in Docker, no API keys. Claude Code skill includ

jshookmcp ★ 2.1K

js hook toolkit that all you need

NewPipeExtractor ★ 2.0K

NewPipe's core library for extracting data from streaming sites

gain ★ 2.0K

Web crawling framework based on asyncio.

article-extractor ★ 1.9K

To extract article from given URL

x-crawl ★ 1.9K

Flexible Node.js AI-assisted crawler library

OpenAdapt ★ 1.9K

For EHRs, payer portals, and any app your team clicks through. A person or an agent shows the task; OpenAdapt turns it into a local program that repeats the ste

browserless ★ 1.8K

The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.

diskover-community ★ 1.8K

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

finvizfinance ★ 1.8K

Finviz analysis python library.

ruia ★ 1.7K

Async Python 3.6+ web scraping micro-framework based on asyncio

selectolax ★ 1.7K

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

single-file-cli ★ 1.7K

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

Scweet ★ 1.7K

Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.

actionbook ★ 1.6K

Let your AI agent get the sources behind logins and paywalls.

PDFLayoutTextStripper ★ 1.6K

Converts a pdf file into a text file while keeping the layout of the original pdf. Useful to extract the content from a table in a pdf file for instance. This i

fredy ★ 1.6K

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt,

Mink ★ 1.6K

PHP web browser emulator abstraction

patchright-python ★ 1.6K

Undetected Python version of the Playwright testing and automation library.

OpenSERP ★ 1.5K

Self-hosted SERP API for Google, Bing, Yandex, and more

wreq-python ★ 1.5K

An ergonomic, privacy-aware Python HTTP Client

agentql ★ 1.5K

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracti

zendriver ★ 1.5K

A blazing fast, async-first, undetectable webscraping/web automation framework based on ultrafunkamsterdam/nodriver. Now with Docker support!

scrapy-playwright ★ 1.5K

🎭 Playwright integration for Scrapy

fscrawler ★ 1.5K

Elasticsearch File System Crawler (FS Crawler)

rebrowser-patches ★ 1.5K

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/

miasma ★ 1.4K

Trap AI web scrapers in an endless poison pit.

WebAI2API ★ 1.4K

WebAI2API: 基于 Camoufox 的网页 AI 转 API 工具，支持 LMArena/Gemini等，多窗口并发与账号隔离。 | Web AI to OpenAI API via Camoufox. Supports LMArena/Gemini and more, multi-window concur

parsera ★ 1.4K

Lightweight library for scraping web-sites with LLMs

httpcloak ★ 1.4K

Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic lev

AIPex ★ 1.3K

AIPex: AI browser automation assistant, no migration and privacy first. Alternative to Manus Browser Operator、 Claude Chrome and Agent Browser

crw ★ 1.2K

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scra

browsertrix-crawler ★ 1.2K

Run a high-fidelity browser-based web archiving crawler in a single Docker container

ClawBench ★ 1.2K

Open-source benchmark for browser AI agents on daily tasks.

newspaper4k ★ 1.2K

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

pdf_oxide ★ 1.2K

The fastest PDF library for Python and Rust. Text extraction, image extraction, markdown conversion, PDF creation & editing. 0.8ms mean, 5× faster than industry

fess ★ 1.1K

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & sema

socid-extractor ★ 1.1K

⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites

faster-than-requests ★ 1.1K

Faster requests on Python 3

trawl ★ 1.1K

Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and dr

wreq ★ 1.1K

An ergonomic, privacy-aware Rust HTTP Client

ovobrowser ★ 1.0K

源码级 Chromium 指纹内核，检测站按普通 Chrome 评分。Source-level Chromium fingerprint kernel that passes bot detection tests. Not JS injection — C++ patches compiled into the bi

opentabs ★ 1.0K

Browser automation clicks buttons. OpenTabs calls APIs.

siteone-crawler ★ 992

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA

x-reader ★ 988

Universal content reader MCP Server for 10+ platforms

scout ★ 948

Scout: the open-source browser agent, by Outcrawl. Mirrored automatically from Outcrawl's own repository.

icrawler ★ 934

A multi-thread crawler framework with many builtin image crawlers provided.

TikHub-API-Python-SDK ★ 931

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱)

auto-browser ★ 915

Give your AI agent a real browser — with a human in the loop. Open-source MCP-native browser agent.

skrape.it ★ 874

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particula

ArrowDL ★ 853

ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux

wa-js ★ 830

Browser-side WhatsApp Web automation library. Exposes the WPP JavaScript API for chats, messages, contacts, groups, media and events; powers WPPConnect integrat

patchright-nodejs ★ 803

Undetected NodeJS version of the Playwright testing and automation library.

js-cookie-monitor-debugger-hook ★ 782

js cookie逆向利器：js cookie变动监控可视化工具 & js cookie hook打条件断点

figranium ★ 780

Stack blocks visually to build complex browser workflows and execute them via API

xxl-crawler ★ 756

A lightweight web crawler framework.（Java爬虫框架）

Prism-Browser-Community ★ 745

Local-first open-source multi-profile fingerprint browser based on Chromium and Electron

TumblThree ★ 742

A Tumblr and Twitter Blog Backup Application

ketch ★ 739

Fast, stateless CLI for web search and scrape. Built for AI agents.

PyPtt ★ 734

The best PTT library

scrapecraft ★ 724

🤖 AI-powered web scraping editor with visual workflow builder. Build, test & deploy web scrapers using natural language. Powered by ScrapeGraphAI & LangGraph.

deepcrawl ★ 695

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.

terminal ★ 659

Terminal UI to get stuff done in the browser

agent-browser-cli ★ 642

使用 agent-browser-cli 进行浏览器感知与控制。适用于标签页扫描/切换、页面 JS 执行、Cookie、CDP、contentSettings、截图、文件上传、下拉框点击、tmwd_cdp_bridge 初始化和 Web 工具排障

PHPScraper ★ 588

A universal web-util for PHP.

freeproxy ★ 571

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)

Stealth-Requests ★ 566

Undetected web-scraping & seamless HTML parsing in Python!

basketball_reference_web_scraper ★ 563

NBA Stats API via Basketball Reference

reader ★ 561

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

oc ★ 547

Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.

browser-search ★ 539

A skill for AI agents: search the web with SearXNG, browse with Camofox, bypass protections with CloakBrowser. Anti-hallucination by design. Self-hosted, free,

ff-ruyijs ★ 505

Run any site JavaScript from Python with the real Firefox engine. Headless SpiderMonkey + Necko for automation and scraping.

invisible_selenium ★ 270

Selenium that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scr

dsh-antibrow ★ 12

DeepSeek Harness plugin: give your agent a browser with a persistent identity - engine-level fingerprint spoofing, unlimited free local profiles, Android device