Crawler v2 : Advanced and Typescript version of node-crawler
Features:
- Server-side DOM & automatic jQuery insertion with Cheerio (default),
- Configurable pool size and retries,
- Control rate limit,
- Priority queue of requests,
- let crawler deal for you with charset detection and conversion,
If you have prior experience with Crawler v1, for fast migration, please proceed to the section Differences and Breaking Changes.
Use with AI agents
This package ships an agent skill that teaches AI agents (e.g. OpenClaw, Hermes, Claude Code, Codex etc) how to crawl with crawler — queue vs. direct requests, rate limiting, proxy rotation, Cheerio parsing, and common pitfalls.
Recommended — install from ClawHub: clawhub.ai/mike442144/node-crawler
openclaw skills install node-crawler
Fallback — download the skill bundle from the latest release and unzip it into your agent's skills directory:
curl -L -o node-crawler-skill.zip https://github.com/bda-research/node-crawler/releases/latest/download/node-crawler-skill.zip
unzip node-crawler-skill.zip -d .openclaw/skills/
Let your agent install it — paste this prompt:
Install the
node-crawlerskill so you can use it for large-scale web crawling: downloadhttps://github.com/bda-research/node-crawler/releases/latest/download/node-crawler-skill.zip, unzip it, and place thenode-crawler/folder in your skills directory (e.g..openclaw/skills/,.claude/skills/, or wherever your runtime loads skills from).
The agent loads it automatically when a task involves large-scale scraping or crawling.
Quick start
Install
Requires Node.js 22 or above.
$ npm install crawler
Warning: Given the dependencies involved (Especially migrating from request to got) , Crawler v2 has been designed as a native ESM and no longer offers a CommonJS export. We would also like to recommend that you convert to ESM. Note that making this transition is generally not too difficult.If you have a large codebase built with Crawler v1, you can upgrade to v2.0.3-beta (using npm install crawler@beta), which supports both ESM and CommonJS builds.Please note that code previously using the "body" parameter to send form data in POST requests will need to be updated to use "form" even in the beta version.
Usage
Execute asynchronously via custom options
import Crawler from "crawler";
const c = new Crawler({
maxConnections: 10,
// This will be called for each crawled page
callback: (error, res, done) => {
if (error) {
console.log(error);
} else {
const $ = res.$;
// $ is Cheerio by default
//a lean implementation of core jQuery designed specifically for the server
console.log($("title").text());
}
done();
},
});
// Add just one URL to queue, with default callback
c.add("http://www.amazon.com");
// Add a list of URLs
c.add(["http://www.google.com/", "http://www.yahoo.com"]);
// Add URLs with custom callbacks & parameters
c.add([
{
url: "http://parishackers.org/",
jQuery: false,
// The global callback won't be called
callback: (error, res, done) => {
if (error) {
console.log(error);
} else {
console.log("Grabbed", res.body.length, "bytes");
}
done();
},
},
]);
// Add some HTML code directly without grabbing (mostly for tests)
c.add([
{
html: "<title>This is a test</title>",
},
]);
please refer to options for detail.
Slow down
Use rateLimit to slow down when you are visiting web sites.
import Crawler from "crawler";
const c = new Crawler({
rateLimit: 1000, // `maxConnections` will be forced to 1
callback: (err, res, done) => {
console.log(res.$("title").text());
done();
},
});
c.add(tasks); //between two tasks, minimum time gap is 1000 (ms)
Custom parameters
Sometimes you have to access variables from previous request/response session, what should you do is passing parameters in options.userParams :
c.add({
url: "http://www.google.com",
userParams: {
parameter1: "value1",
parameter2: "value2",
parameter3: "value3",
},
});
then access them in callback via res.options
console.log(res.options.userParams);
Raw body
If you are downloading files like image, pdf, word etc, you have to save the raw response body which means Crawler shouldn't convert it to string. To make it happen, you need to set encoding to null
import Crawler from "crawler";
import fs from "fs";
const c = new Crawler({
encoding: null,
jQuery: false, // set false to suppress warning message.
callback: (err, res, done) => {
if (err) {
console.error(err.stack);
} else {
fs.createWriteStream(res.options.userParams.filename).write(res.body);
}
done();
},
});
c.add({
url: "https://raw.githubusercontent.com/bda-research/node-crawler/master/crawler_primary.png",
userParams: {
filename: "crawler.png",
},
});
preRequest
If you want to do something either synchronously or asynchronously before each request, you can try the code below. Note that direct requests won't trigger preRequest.
import Crawler from "crawler";
const c = new Crawler({
preRequest: (options, done) => {
// 'options' here is not the 'options' you pass to 'c.queue', instead, it's the options that is going to be passed to 'request' module
console.log(options);
// when done is called, the request will start
done();
},
callback: (err, res, done) => {
if (err) {
console.log(err);
} else {
console.log(res.statusCode);
}
},
});
c.add({
url: "http://www.google.com",
// this will override the 'preRequest' defined in crawler
preRequest: (options, done) => {
setTimeout(() => {
console.log(options);
done();
}, 1000);
},
});
Direct request
Support both Promise and callback
import Crawler from "crawler";
const crawler = new Crawler();
// When using directly "send", the preRequest won't be called and the "Event:request" won't be triggered
const response = await crawler.send("https://github.com/");
console.log(response.options);
// console.log(response.body);
crawler.send({
url: "https://github.com/",
// When calling `send`, `callback` must be defined explicitly, with two arguments `error` and `response`
callback: (error, response) => {
if (error) {
console.error(error);
} else {
console.log("Hello World!");
}
},
});