SiteOne Crawler
SiteOne Crawler is a powerful and easy-to-use website analyzer, cloner, and converter designed for developers seeking security and performance insights, SEO specialists identifying optimization opportunities, and website owners needing reliable backups and offline versions.
Now rewritten in Rust for maximum performance, minimal resource usage, and zero runtime dependencies. The transition from PHP+Swoole to Rust resulted in 25% faster execution and 30% lower memory consumption while producing identical output.
Discover the SiteOne Crawler advantage:
- Run Anywhere: Single native binary for 🪟 Windows, 🍎 macOS, and 🐧 Linux (x64 & arm64). No runtime dependencies.
- Work Your Way: Launch the binary without arguments for an interactive wizard 🧙 with 10 preset modes, use the extensive command-line interface 📟 (releases, ▶️ video) for automation and power, or enjoy the intuitive desktop GUI application 💻 (GUI app, ▶️ video) for visual control.
- Rich Output Formats: Interactive HTML audit report 📊 with sortable tables and quality scoring (0.0-10.0) (see nextjs.org sample), detailed JSON for programmatic consumption, and human-readable text for terminal. Send HTML reports directly to your inbox via built-in SMTP mailer 📧.
- CI/CD Integration: Built-in quality gate (
--ci) with configurable thresholds — exit code 10 on failure enables automated deployment blocking. Also useful for cache warming — crawling the entire site after deployment populates your reverse proxy/CDN cache. - Offline & Markdown Power: Create complete offline clones 💾 for browsing without a server (nextjs.org clone) or convert entire websites into clean Markdown 📝 — perfect for backups, documentation, or feeding content to AI models (examples).
- Deep Crawling & Analysis: Thoroughly crawl every page and asset, identify errors (404s, redirects), generate sitemaps 🗺️, and even get email summaries 📧 (watch ▶️ video example).
- Learn More: Dive into the 🌐 Project Website, explore the detailed Documentation, or check the JSON/Text output specs.
GIF animation of the crawler in action (also available as a ▶️ video):
Table of contents
- ✨ Features
- 🚀 Installation
- ▶️ Usage
- Interactive wizard
- Basic example
- URL list example
- CI/CD example
- Fully-featured example
- ⚙️ Arguments
- Basic settings
- Output settings
- Resource filtering
- Advanced crawler settings
- File export settings
- Mailer options
- Upload options
- Offline exporter options
- Markdown exporter options
- Sitemap options
- Expert options
- Fastest URL analyzer
- SEO and OpenGraph analyzer
- Slowest URL analyzer
- Built-in HTTP server
- HTML-to-Markdown conversion
- CI/CD settings
- 🤖 AI assistant (optional)
- 🌐 Browser rendering (optional)
- 🏆 Quality Scoring
- 🔄 CI/CD Integration
- 📄 Output Examples
- 🧪 Testing
- ⚠️ Disclaimer
- 📜 License
✨ Features
In short, the main benefits can be summarized in these points:
- 🕷️ Crawler - very powerful crawler of the entire website reporting useful information about each URL (status code, response time, size, custom headers, titles, etc.)
- 🛠️ Dev/DevOps assistant - offers stress/load testing with configurable concurrent workers (
--workers) and request rate (--max-reqs-per-sec), cache warming, localhost testing, and rich URL/content-type filtering - 📊 Analyzer - analyzes all webpages and reports strange or error behaviour and useful statistics (404, redirects, bad practices, SEO and security issues, heading structures, etc.)
- 📧 Reporter - interactive HTML audit report, structured JSON, and colored text output; built-in SMTP mailer sends HTML reports directly to your inbox
- 💾 Offline website generator - clone entire websites to browsable local HTML files (no server needed) including all assets. Supports multi-domain clones — include subdomains or external domains with intelligent cross-linking.
- 📝 Website to markdown converter - export the entire website to browsable text markdown (viewable on GitHub or any
text editor), or generate a single-file markdown with smart header/footer deduplication — ideal for feeding to AI
tools. Includes a built-in web server that renders markdown exports as styled HTML pages.
Also supports standalone HTML-to-Markdown conversion of local files (
--html-to-markdown). See markdown examples. - 🗺️ Sitemap generator - allows you to generate
sitemap.xmlandsitemap.txtfiles with a list of all pages on your website - 🏆 Quality scoring - automatic quality scoring (0.0-10.0) across 5 categories: Performance, SEO, Security, Accessibility, Best Practices
- 🔄 CI/CD quality gate - configurable thresholds with exit code 10 on failure for automated pipelines; also useful as a post-deployment cache warmer for reverse proxies and CDNs
The following features are summarized in greater detail:
🕷️ Crawler
- all major platforms supported without dependencies (🐧 Linux, 🪟 Windows, 🍎 macOS, arm64) — single native binary
- has incredible 🚀 native Rust performance with async I/O and multi-threaded crawling
- provides simulation of different device types (desktop/mobile/tablet) thanks to predefined User-Agents
- will crawl all files, styles, scripts, fonts, images, documents, etc. on your website
- will respect the
robots.txtfile and will not crawl the pages that are not allowed - has a beautiful interactive and 🎨 colourful output
- it will clearly warn you ⚠️ of any wrong use of the tool (e.g. input parameters validation or wrong permissions)
- as
--urlparameter, you can specify also asitemap.xmlfile (or sitemap index), which will be processed as a list of URLs. In sitemap-only mode, the crawler follows only URLs from the sitemap — it does not discover additional links from HTML pages. Gzip-compressed sitemaps (*.xml.gz) are fully supported, both as direct URLs and when referenced from sitemap index files. - with
--url-list=you can crawl a bounded list of URLs fr