Open source spider projects
Every project in the registry tagged spider, ranked by real GitHub adoption.
Elegant Scraper and Crawler Framework for Golang
Python ProxyPool for web spider
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Every web site provides APIs.
Foundational low latency web data collecting in Rust
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Web Scraping Framework
Web crawling framework based on asyncio.
Flexible Node.js AI-assisted crawler library
Async Python 3.6+ web scraping micro-framework based on asyncio
A multi-thread crawler framework with many builtin image crawlers provided.
A lightweight web crawler framework.（Java爬虫框架）
FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)
Related tags
Frequently asked questions
How many open source spider projects are there?
This registry tracks 14 projects tagged spider, with 87,961 GitHub stars between them. The most-adopted is colly at 25,572 stars.
Are these spider projects free to use?
Yes — 14 of the 14 carry an explicit open-source licence across 4 distinct licences, so there is no licence fee. Where a project also sells a hosted or enterprise version, the self-hosted path remains free.
Which spider project should I choose?
The list above is ranked by GitHub stars, but stars measure attention rather than fit. Check three things on each card: the licence (permissive versus copyleft), the language it is written in, and the last-push date — a high-star project that has not been pushed in a year is a liability.
Are these spider projects still maintained?
8 of the 14 were pushed in the last 90 days, and every card shows its exact last-push date so you can see the rest. Sort your shortlist by that date before committing to a migration.