Open source crawler projects

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDce

LLM-ready web crawler built for AI data pipelines

Scrapy, a fast high-level web crawling & scraping framework for Python.

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

Python scraper based on AI

👾 Fast and simple video download library and CLI tool written in Go

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or G

Elegant Scraper and Crawler Framework for Golang

Python ProxyPool for web spider

🚀 Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for posts, profiles, comments and pla

No-code web scraping, crawling, and extraction platform

A next-generation crawling and spidering framework.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A scalable web crawler framework for Java.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, P

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Headless Chrome .NET API

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Every web site provides APIs.

All In One Web Recon

Node.js scraper to get data from Google Play

Foundational low latency web data collecting in Rust

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Web Scraping Framework

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamste

NewPipe's core library for extracting data from streaming sites

Web crawling framework based on asyncio.

Flexible Node.js AI-assisted crawler library

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

Async Python 3.6+ web scraping micro-framework based on asyncio

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt,

An ergonomic, privacy-aware Python HTTP Client

Elasticsearch File System Crawler (FS Crawler)

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scra

Run a high-fidelity browser-based web archiving crawler in a single Docker container

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & sema

An ergonomic, privacy-aware Rust HTTP Client

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA

A multi-thread crawler framework with many builtin image crawlers provided.

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱)

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particula

ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux

Open source SEO audit tool.

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities usi

js cookie逆向利器：js cookie变动监控可视化工具 & js cookie hook打条件断点

A lightweight web crawler framework.（Java爬虫框架）

A Tumblr and Twitter Blog Backup Application

The best PTT library

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.