Browsertrix Crawler is a standalone, browser-based, high-fidelity web archiving crawler that runs a complex, customizable crawl in a single Docker container, built for web archivists, digital preservation teams, and developers who need faithful captures of browser-rendered pages.

What it is

Browsertrix Crawler is an open-source crawling system written in TypeScript and licensed under AGPL-3.0 or later. It uses Puppeteer to control one or more Brave Browser windows in parallel, and it captures data through the Chrome DevTools Protocol (CDP) inside the browser. The whole system is packaged to run as a standalone crawl inside a single Docker container, which means the browser, the automation layer, and the capture pipeline ship together rather than being assembled by the operator. It lives in the Webrecorder ecosystem, and its topics place it squarely in web archiving, with WARC and WACZ as the archive formats it works with.

The concrete problem it solves is capturing pages that only exist after a browser has executed their JavaScript. Ordinary request-based crawlers see initial HTML and miss content rendered later, so interactive or script-heavy sites are archived incompletely. Browsertrix Crawler drives a real browser instead, and records what the browser actually loaded. Its functionality was originally developed to support the zimit project, in a collaboration between Webrecorder and Kiwix, and the project was then split off from Zimit into a core component of Webrecorder. In that sense, it replaces the arrangement where this browser-based capture capability lived inside Zimit itself. Initial support for the 0.x line came from Kiwix, and Portico provided additional support including for the 0.4.x version.

Key capabilities

• Uses Puppeteer to control one or more Brave Browser windows in parallel during a crawl.
• Captures page data through the Chrome DevTools Protocol (CDP) from inside the browser.
• Runs as a standalone system inside a single Docker container, with no separate browser fleet to provision.
• Handles complex and customizable crawl configurations rather than a fixed one-size crawl.
• Works with the WARC and WACZ web archive formats, as reflected in the project topics.
• Implements the crawler in TypeScript and releases it under AGPL-3.0 or later.
• Maintains hosted documentation at crawler.docs.browsertrix.com.

Who uses it and how

• Web archivists and digital preservation teams use it to capture sites whose content depends on browser execution.
• The zimit project, developed by Webrecorder and Kiwix, relies on this crawler for its browser-based capture step.
• It serves as a core component of Webrecorder, so Webrecorder-based workflows inherit its capture behaviour.
• Operators run it as a single container, which suits environments where spinning up a managed browser cluster is impractical.
• Parallel Brave Browser windows let a single crawl cover more of a site at once than a strictly sequential browser session would.

Getting started

The documented deployment method is to run Browsertrix Crawler as a standalone Docker container; setup and usage instructions are hosted at crawler.docs.browsertrix.com. Build and development documentation is available in the repository's docs section.

How it compares

No list of paid products that this project replaces is provided in the available facts, so a direct commercial comparison cannot be made. Among the tools named in those facts, the closest relative is zimit: Browsertrix Crawler was split off from Zimit and became a core Webrecorder component, while Zimit remains the project it was originally built to support. Both sit in the same open web-archiving space, and the crawler is the browser-based capture engine that the split produced.

When to use it — and when not to

A self-hoster must run and maintain a Docker container that includes browser automation, so the operational surface is larger than a plain HTTP crawler's. Teams that only need lightweight request-based fetching, or that cannot accept the copyleft terms of AGPL-3.0 in their distribution model, should look elsewhere. The repository also carries 145 open issues and a fairly sparse README, with the substantive guidance pushed out to the external documentation site, so anyone expecting a self-contained quickstart in the repository should plan for that.