Open source crawling projects
Every project in the registry tagged crawling, ranked by real GitHub adoption.
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDce
Scrapy, a fast high-level web crawling & scraping framework for Python.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or G
Elegant Scraper and Crawler Framework for Golang
No-code web scraping, crawling, and extraction platform
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, P
A Chrome DevTools Protocol driver for web automation and scraping.
Headless Chrome .NET API
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Web Scraping Framework
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/
Run a high-fidelity browser-based web archiving crawler in a single Docker container
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA
Open source SEO audit tool.
100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.
Related tags
Frequently asked questions
How many open source crawling projects are there?
This registry tracks 15 projects tagged crawling, with 259,223 GitHub stars between them. The most-adopted is Scrapling at 91,885 stars.
Are these crawling projects free to use?
Yes — 15 of the 15 carry an explicit open-source licence across 5 distinct licences, so there is no licence fee. Where a project also sells a hosted or enterprise version, the self-hosted path remains free.
Which crawling project should I choose?
The list above is ranked by GitHub stars, but stars measure attention rather than fit. Check three things on each card: the licence (permissive versus copyleft), the language it is written in, and the last-push date — a high-star project that has not been pushed in a year is a liability.
Are these crawling projects still maintained?
14 of the 15 were pushed in the last 90 days, and every card shows its exact last-push date so you can see the rest. Sort your shortlist by that date before committing to a migration.