WebsiteBench is a free, open source data extraction & web scraping project written in Python and released under Apache-2.0. It has 525 GitHub stars, 48 forks and 91 open issues, and was last pushed 31 hours ago. On this registry it ranks #154 of 156 tracked projects in Data Extraction & Web Scraping, with 5 head-to-head comparisons available.

What is WebsiteBench?

WebsiteBench is an Apache-2.0 Python benchmark that asks whether AI agents can rebuild a production website as a full-stack application by exploring it through a browser, and it is aimed at agent and LLM researchers who need a hard, verifiable measure of end-to-end building ability.

What it is

WebsiteBench lives in the LLM agent evaluation ecosystem. Given a production-level website that they can explore through a browser, agents must reverse-engineer a full-stack version that matches the original's appearance and behaviour as closely as possible. The suite consists of 70 websites and 46,916 hidden test cases, distributed through Harbor Hub as websitebench/WebsiteBench and through Hugging Face as TIGER-Lab/WebsiteBench. A PyPI package, websitebench, handles downloading tasks, and execution is delegated to the Harbor framework, which the project pins at version 0.25.0.

The concrete problem it solves is the absence of a repeatable, quantitative verdict on agent-built websites. Rather than judging a generated site by eye, WebsiteBench scores each task automatically: every task carries between 302 and 2,016 hidden test cases, and a case passes only when its functional checks pass and every visual checkpoint reaches a normalized SSIM of 0.99. A task's reward is the share of its cases that pass, written to jobs/.../verifier/reward.txt, so results across agents and models are directly comparable.

Key capabilities

  • 70 task websites with 46,916 hidden test cases, published on Harbor Hub (websitebench/WebsiteBench@v1) and Hugging Face (TIGER-Lab/WebsiteBench).
  • A command-line downloader in the websitebench package, for example websitebench download --task medium, for fetching task definitions locally.
  • Dataset-wide or single-task runs through Harbor, using --dataset websitebench/WebsiteBench@v1 for all 70 tasks or --task websitebench/medium@v1 for one.
  • Swappable agents and models at run time, as in --agent codex --model ' ', with concurrency controlled by --n-concurrent.
  • A dual scoring model in which a test case requires both passing functional checks and a normalized SSIM of at least 0.99 at every visual checkpoint.
  • An oracle reference solution per task, run with harbor run --task websitebench/medium@v1 --agent oracle, for validating a task before evaluating an agent.
  • Explicit network allowlisting for sandboxed runs, covering hosts such as snapshot.debian.org, raw.githubusercontent.com, github.com, nodejs.org and registry.npmjs.org.

Who uses it and how

  • Benchmark researchers evaluating browser-using coding agents on full-stack reconstruction, where the deliverable is a working site rather than a patch or a snippet.
  • Teams comparing agents or models on a like-for-like basis by holding the dataset fixed and varying only --agent and --model.
  • Groups working at small local scale, using --n-concurrent 1 on a single machine, or at larger scale by following the cloud instructions in docs/cloud.md.
  • Groups tracking published outcomes against the leaderboard hosted at website-bench.com/#results, alongside the task list in docs/tasks.md.

Getting started

Task data is installed with pip install websitebench, while the benchmark itself requires Docker plus Harbor installed via uv tool install harbor==0.25.0, after which harbor run --dataset websitebench/WebsiteBench@v1 executes an agent across all 70 tasks.

How it compares

The facts provided here name no paid products that WebsiteBench replaces, and the only adjacent tool named is the Harbor framework, which is the harness it runs on rather than a peer benchmark. It therefore stands alone in this registry as a website-reconstruction benchmark, and any comparison with other agent evaluations is not supported by the available information.

When to use it — and when not to

A self-hoster must operate Docker, keep Harbor pinned at 0.25.0, supply an API key such as OPENAI_API_KEY, and manage the network allowlist for sandboxed hosts, so it is not a drop-in library. Despite its placement in the data extraction and web scraping category, it is an evaluation benchmark rather than a scraper and should not be chosen by anyone seeking extraction tooling. Two limitations are worth noting: the reference websites reproduce third-party sites for non-commercial research only, and the task definitions carry a harbor-canary GUID that must be excluded from training data.

project readme (upstream, from github) — read inline

Project Page Leaderboard Tasks Harbor Hub Paper

Quickstart · Tasks · Run in the cloud · Docs · PyPI · Citation

Given a production-level website they can explore through a browser, agents must reverse-engineer a full-stack version that matches its appearance and behavior as closely as possible.

Tasks

70 websites and 46,916 hidden test cases, on Harbor Hub and Hugging Face. See the task list.

Download them with the PyPI package:

pip install websitebench
websitebench download --task medium

Running the benchmark

Install Docker and Harbor:

uv tool install harbor==0.25.0

Run an agent on all 70 tasks:

export OPENAI_API_KEY='<key>'
harbor run --dataset websitebench/WebsiteBench@v1 \
  --agent codex --model '<model>' --n-concurrent 1 \
  --allow-environment-host snapshot.debian.org \
  --allow-environment-host raw.githubusercontent.com \
  --allow-environment-host github.com \
  --allow-environment-host nodejs.org \
  --allow-environment-host registry.npmjs.org

To run one task, replace --dataset websitebench/WebsiteBench@v1 with --task websitebench/medium@v1.

Check a task with its reference solution:

harbor run --task websitebench/medium@v1 --agent oracle

Each trial's reward is in jobs///verifier/reward.txt. The docs cover other agents, downloads and cloud machines.

Scoring

Each task has 302 to 2,016 hidden test cases. A case passes when its functional checks pass and every visual checkpoint reaches a normalized SSIM of 0.99. A task's reward is the share of its cases that pass.

Citation

@misc{zhang2026websitebench,
  title  = {WebsiteBench: Can AI Agents Rebuild Websites through Browser Exploration?},
  author = {{WebsiteBench Authors}},
  year   = {2026},
  url    = {https://github.com/TIGER-AI-Lab/WebsiteBench},
}

License

The benchmark code, task definitions, and verifier are released under Apache-2.0. The reference websites reproduce third-party sites for evaluation; their content belongs to its owners and is provided for non-commercial research only.

The task definitions carry a harbor-canary GUID. Please exclude documents that contain it from training data.

Star History

Star History Chart

Frequently asked questions

Is WebsiteBench free to use?

WebsiteBench is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does WebsiteBench do?

Can AI agents rebuild websites through browser exploration?

What is WebsiteBench written in?

WebsiteBench is primarily written in Python. Its source is publicly available at https://github.com/TIGER-AI-Lab/WebsiteBench, and it has 525 GitHub stars.