XXL-CRAWLER is a lightweight, open-source Java web crawler framework that lets a developer start a multi-threaded crawler with a single line of code and map collected page data into Java objects through annotations, built for Java teams that need to gather web data without adopting a heavyweight scraping platform.

What it is

XXL-CRAWLER is a crawler framework written in Java and published under the Apache License, Version 2.0, by xuxueli. Its stated design goal is a "lightweight web crawler framework" whose underlying implementation depends strongly on only one library, jsoup, and whose feature list covers multi-threaded execution, fully annotated extraction, JS rendering, proxy support, and distributed extension. The project lives in the Java ecosystem and is distributed as a library that a developer embeds in an application, rather than as a standalone crawling service or hosted product. It carries 758 stars, 318 forks, and 7 open issues, and its source repository is actively maintained, with the last push recorded on 2026-08-08.

The concrete problem it solves is the boilerplate that surrounds routine page collection in Java. Instead of hand-writing thread pools, retry loops, duplicate suppression, and per-field parsing logic, a developer annotates a plain Java class such as a PageVO, and the framework performs the extraction and object assembly automatically. It replaces the ad-hoc combination of an HTTP client, an HTML parser, and a custom scheduling layer that teams would otherwise assemble themselves, and it provides a documented path from a single-machine crawl to a distributed one when the volume of URLs grows beyond what one process can handle.

Key capabilities

• Annotated page extraction: page data maps to PageVO objects via annotations, and a single page can yield one or more PageVO instances without hand-written parsing code.
• Pluggable page loading through the PageLoader module, which ships with a native Jsoup implementation for speed and a Selenium + ChromeDriver implementation for JavaScript-rendered content, and accepts further custom implementations.
• Distributed crawling by extending the RunUrlPool module and sharing run data through Redis or a database, while a LocalRunUrlPool single-machine pool is provided by default.
• Proxy IP support to counter anti-collection WAF rules, with runtime adjustment of the proxy pool and custom proxy pool routing strategies.
• Idempotent deduplication that prevents pages from being crawled twice, plus URL diffusion filtering through configurable page whitelist regular expressions.
• Failure retry with configurable retry counts, request timeout control, and deliberate pauses after page processing to reduce the chance of being blocked.
• Custom request information including request parameters, Cookie, Header, rotating UserAgent, and Referrer, all of which can be adjusted dynamically at runtime.

Who uses it and how

• Teams extracting paginated article listings, as in the sample crawler that diffuses across pages of CNBlog digest articles and annotates the extracted fields into PageVO output.
• Data collection jobs that need raw HTML rather than parsed fields, such as downloading source pages to local HTML files.
• Media and image harvesting workflows, illustrated by a sample that downloads images from NetEase news articles to local storage.
• Non-web interface collection, demonstrated by a sample that calls a JSON endpoint, such as Baidu news, and outputs the response data directly.
• Price monitoring against JavaScript-rendered storefronts, where the sample uses Selenium + ChromeDriver to simulate browser behaviour because prices are rendered asynchronously.
• Larger crawls run as a cluster, as in the Redis cluster sample where several XxlCrawler instances share one RunUrlPool, cooperate on URL diffusion, and consume pending tasks together.

Getting started

The README does not publish a Maven coordinate, Docker image, or compose file; it directs readers to the example test cases under src/test/java/com/xxl/crawler/test in the repository, and to the Chinese documentation and project home page at xuxueli.com/xxl-crawler for installation and usage detail.

How it compares

No list of paid products that this project replaces is provided in the available facts, and no competing crawler framework is named either. The only comparable tooling named is infrastructure the project itself uses or integrates: it is built on jsoup for fast non-JS parsing, and it offers Selenium with ChromeDriver as an alternative loader when rendering compatibility matters more than speed.

When to use it — and when not to

A self-hoster must operate the surrounding infrastructure the framework depends on, which means Redis or a database for distributed run data, a Selenium and ChromeDriver environment if JavaScript rendering is required, and a proxy pool if the target sites apply WAF rules. It is a poor fit for teams that need a managed service, a graphical crawling console, or Python-based tooling, since this is a Java library embedded in application code. The main limitation visible in the facts is documentation depth: the README is largely Chinese, the English surface is thin, and the repository explicitly presents its test code as examples for learning and testing rather than as production guidance, so a serious deployment depends on reading the source and the external documentation site.