Open source scraping projects
Every project in the registry tagged scraping, ranked by real GitHub adoption.
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python scraper based on AI
Elegant Scraper and Crawler Framework for Golang
A scalable web crawler framework for Java.
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
An open source fingerprint browser based on Ungoogled Chromium. 指纹浏览器 隐私浏览器
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
Foundational low latency web data collecting in Rust
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Web Scraping Framework
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracti
Lightweight library for scraping web-sites with LLMs
📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.
⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites
Fast, stateless CLI for web search and scrape. Built for AI agents.
A universal web-util for PHP.
Related tags
Frequently asked questions
How many open source scraping projects are there?
This registry tracks 18 projects tagged scraping, with 363,335 GitHub stars between them. The most-adopted is firecrawl at 197,942 stars.
Are these scraping projects free to use?
Yes — 18 of the 18 carry an explicit open-source licence across 7 distinct licences, so there is no licence fee. Where a project also sells a hosted or enterprise version, the self-hosted path remains free.
Which scraping project should I choose?
The list above is ranked by GitHub stars, but stars measure attention rather than fit. Check three things on each card: the licence (permissive versus copyleft), the language it is written in, and the last-push date — a high-star project that has not been pushed in a year is a liability.
Are these scraping projects still maintained?
14 of the 18 were pushed in the last 90 days, and every card shows its exact last-push date so you can see the rest. Sort your shortlist by that date before committing to a migration.