[opensourcedex]

browsertrix-crawler is a free, open source data extraction & web scraping project written in TypeScript and released under AGPL-3.0. It has 1,140 GitHub stars, 150 forks and 145 open issues, and was last pushed 2 hours ago. On this registry it ranks #71 of 83 tracked projects in Data Extraction & Web Scraping, with 5 head-to-head comparisons available.

Browsertrix Crawler 1.x

Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses Puppeteer to control one or more Brave Browser browser windows in parallel. Data is captured through the Chrome Devtools Protocol (CDP) in the browser.

For information on how to use and develop Browsertrix Crawler, see the hosted Browsertrix Crawler documentation.

For information on how to build the docs locally, see the docs page.

Support

Initial support for 0.x version of Browsertrix Crawler, was provided by Kiwix. The initial functionality for Browsertrix Crawler was developed to support the zimit project in a collaboration between Webrecorder and Kiwix, and this project has been split off from Zimit into a core component of Webrecorder.

Additional support for Browsertrix Crawler, including for the development of the 0.4.x version has been provided by Portico.

License

AGPLv3 or later, see LICENSE for more details.

Frequently asked questions

Is browsertrix-crawler free to use?

browsertrix-crawler is open source under the AGPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does browsertrix-crawler do?

Run a high-fidelity browser-based web archiving crawler in a single Docker container

What is browsertrix-crawler written in?

browsertrix-crawler is primarily written in TypeScript. Its source is publicly available at https://github.com/webrecorder/browsertrix-crawler, and it has 1,140 GitHub stars.