reverse-api-engineer: Open Source Data Extraction & Web Scraping

What is reverse-api-engineer?

Reverse API Engineer is an MIT-licensed Python CLI and agent that turns a website into a typed API client by capturing the site's own network traffic and having a language model write the client from it, built for developers who need programmatic access to a site that offers no usable public API.

What it is

Reverse API Engineer lives in the Python CLI ecosystem and drives browser automation and AI agents to reverse-engineer web endpoints. The user supplies a website and a goal, a browser visits the site either under manual control or under an autonomous agent, network traffic is captured to a HAR file, and the configured model reads that traffic and writes a working API client. It sits alongside browser tooling such as Playwright MCP, the Chrome DevTools MCP server, and the Vercel agent-browser CLI, which it uses for capture rather than replacing.

The concrete thing it replaces is the manual reverse-engineering loop: opening DevTools, copying cURL commands, and gluing together a client by hand. Instead of that, the tool reads the recorded sessions and emits a finished, typed client that talks to the endpoints the site actually uses. It supports agent mode, where the model browses on its own, and manual mode, where the developer drives the browser and only generation is automated.

Key capabilities

Four modes cycled with Shift+Tab: manual (developer drives the browser), agent (AI agent drives capture), engineer (re-run generation on a previous capture), and collector (agent gathers structured data as JSON or CSV using web search and fetch).
Generates clients in ten output languages: Python, JavaScript, TypeScript, Go, Java, C#, PHP, Ruby, and C.
Writes a runnable output folder such as ./scripts/apple_jobs_api/ containing api_client.py, README.md, and example_usage.py.
Three agent providers: auto (Playwright MCP), chrome-mcp (drives the real Chrome so existing sessions and cookies are kept, requires Chrome 146+ and Node.js 20.19+), and agent-browser (the Vercel agent-browser CLI).
Settings live in ~/.reverse-api/config.json and are editable through /settings in the CLI, covering agent_provider, output_language, output_dir, real_time_sync, and sdk.
Selectable SDK and model backends: claude, opencode, cursor, or copilot, with Sonnet 4.6 as default, Opus 4.6 as most capable, and Haiku 4.5 as fastest.
Optional local model servers through Ollama at http://127.0.0.1:11434 and OpenCode at http://127.0.0.1:4096, where OpenCode is started or reused via npx without a global install.

Who uses it and how

Data extraction and scraping engineers who currently hand-build clients from DevTools sessions and want the capture-to-client step automated.
Teams targeting authenticated sites, who use the chrome-mcp provider so the agent inherits existing Chrome sessions and cookies rather than logging in again.
Developers with privacy or cost constraints, who point ollama_base_url at a local Ollama instance or run OpenCode locally instead of sending captured traffic to a hosted provider.
Groups standardized on a particular coding assistant, choosing the matching sdk value of claude, opencode, cursor, or copilot.
Trimmed Linux and CI environments, where the first Chromium fetch uses agent-browser install with --with-deps.

Getting started

Install with uv tool install reverse-api-engineer or pip install reverse-api-engineer, then run reverse-api-engineer and enter a goal such as "fetch all apple jobs from their careers page". Manual mode needs the optional extra installed as uv tool install "reverse-api-engineer[manual]" followed by playwright install chromium.

How it compares

The facts name no paid products that this project replaces, so the honest placement is among the tools it composes with. It sits above Playwright MCP, the Chrome DevTools MCP server, and the Vercel agent-browser CLI, which handle browsing and traffic capture while Reverse API Engineer adds the model-driven client generation on top. Its sdk setting also makes it a consumer of Claude, OpenCode, Cursor, and GitHub Copilot rather than a competitor to them.

When to use it — and when not to

A self-hoster must supply model access, either a hosted backend such as Claude, OpenCode, Cursor, or GitHub Copilot, or a locally running Ollama or OpenCode server, and agent mode additionally depends on Node.js plus npx-launched browser tooling and a Chromium fetch. Anyone unwilling to run those dependencies, or who needs a guaranteed-stable client without reviewing model-written code before production use, should look elsewhere. The README excerpt provided is also cut off mid-sentence, so the repository should be checked for current setup details; the project does show recent activity, with the last push on 2026-08-30 and 12 open issues.

project readme (upstream, from github) — read inline

Reverse API Engineer

Turn websites into APIs. Browse (or let an agent browse), and get a clean, typed client for the endpoints the site actually uses.

Agent Mode Demo
Agent mode

Manual Mode Demo
Manual mode

How it works

You give it a website and a goal ("fetch all Apple jobs").
A browser visits the site, either driven by you or by an AI agent.
Network traffic is captured to a HAR file.
Your configured model reads the traffic and writes a working API client in Python, JavaScript, TypeScript, Go, Java, C#, PHP, Ruby, or C.

No more manually opening DevTools, copying cURL commands, and gluing together a client.

Install

uv tool install reverse-api-engineer   # or: pip install reverse-api-engineer

Agent mode (the default) captures through npx-launched browser tooling — browser MCP servers (Playwright or Chrome DevTools) or the Vercel agent-browser CLI — and needs no extra Python packages. Manual mode drives a local Playwright browser, which ships in the optional [manual] extra:

uv tool install "reverse-api-engineer[manual]"   # or: pip install "reverse-api-engineer[manual]"
playwright install chromium

Quick start

reverse-api-engineer
> fetch all apple jobs from their careers page

# Browser opens. Navigate, interact, close when done.
# → ./scripts/apple_jobs_api/  (api_client.py, README.md, example_usage.py)

Cycle modes with Shift+Tab:

Mode	What it does
`manual`	You drive the browser; AI generates the client from captured traffic.
`agent`	An AI agent drives capture autonomously (Playwright or Chrome MCP, or Vercel agent-browser CLI).
`engineer`	Re-run generation on a previous capture (`engineer` ).
`collector`	Agent collects structured data (JSON/CSV) using web search + fetch.

Agent mode providers:

auto (default): Playwright MCP, single workflow for browsing + reverse engineering.
chrome-mcp: drives your real Chrome so you keep existing sessions/cookies. Requires Chrome 146+ and Node.js 20.19+.
agent-browser: Vercel agent-browser CLI (not a Reverse API Engineer browser MCP server). At session start RAE uses whatever agent-browser is already on PATH, otherwise runs npm install -g (same pin as config / RAE_AGENT_BROWSER_PACKAGE), prints a yellow notice, validates with --help, and only then falls back to npx -y if npm cannot install. Prompts embed the resolved shell prefix alongside skills get core --full, skills list, HAR phases, cloud notes from agent_browser_notes. Tune with agent_browser_npx_package (optional), env RAE_AGENT_BROWSER_*. First Chromium fetch: agent-browser install (add --with-deps on trimmed Linux).

Optional sanity checks:

agent-browser doctor --offline --quick || true
agent-browser skills list >/dev/null

Configuration

Settings live in ~/.reverse-api/config.json and can be edited via /settings in the CLI:

{
  "agent_provider": "auto",
  "agent_browser_npx_package": "agent-browser@0",
  "agent_browser_notes": "",
  "claude_code_model": "claude-sonnet-4-6",
  "collector_model": "claude-sonnet-4-6",
  "ollama_auto_start": true,
  "ollama_base_url": "http://127.0.0.1:11434",
  "opencode_auto_start": true,
  "opencode_base_url": "http://127.0.0.1:4096",
  "opencode_model": "big-pickle",
  "opencode_npx_package": "opencode-ai@latest",
  "opencode_provider": "opencode",
  "copilot_model": "gpt-5",
  "cursor_model": "composer-2.5",
  "output_dir": null,
  "output_language": "python",
  "real_time_sync": true,
  "sdk": "claude"
}

Models: Sonnet 4.6 (default), Opus 4.6 (most capable), Haiku 4.5 (fastest). For OpenCode see models.dev.
SDK: claude (default), opencode, cursor, or copilot (GitHub Copilot).
OpenCode setup: with sdk: "opencode", RAE reuses an existing server or downloads/starts opencode-ai@latest through npx; a global OpenCode installation is not required. Fresh configurations default to the free opencode/big-pickle model. /settings shows a loading spinner, then offers a searchable OpenCode Provider / Model picker populated from the server's connected, tool-capable catalog and marks free OpenCode options. Before creating a session, RAE validates the saved pair again and suggests currently available free models when configuration is invalid. Compatible older servers are reused with a version warning. Node.js 20+ is required for automatic startup. Password-protected servers use OPENCODE_SERVER_PASSWORD and optional OPENCODE_SERVER_USERNAME. Override startup with OPENCODE_BASE_URL, RAE_OPENCODE_PACKAGE, or RAE_OPENCODE_AUTO_START=0.
Ollama through OpenCode: choose provider ollama in /settings; RAE starts an installed Ollama daemon if needed, lists only installed models with tool calling and 64k+ context, and supplies OpenCode's provider config inline. Models are never downloaded silently. Override with RAE_OLLAMA_BASE_URL or RAE_OLLAMA_AUTO_START=0.
Output language: python, javascript, typescript, go, java, csharp, php, ruby, or c. C needs a POSIX toolchain (cc, libcurl headers) — macOS/Linux, or WSL/MSYS2 on Windows.

CLI

Slash commands inside the CLI:

/settings: configure model, SDK, agent provider, and sync settings.
/history: list past runs with timestamps, costs, and status.
/messages : view detailed message logs for a run.
/help (alias: /commands): show the command list.
/exit (alias: /quit): leave the CLI.

Scriptable subcommands (pipe to jq):

reverse-api-engineer agent --prompt "capture the public jobs api" \
  --url https://example.com/jobs --json | jq

reverse-api-engineer list --json
reverse-api-engineer show <run_id> --json
reverse-api-engineer run <run_id> --file api_client.py \
  --no-interactive --auto-install -- --org acme

Pass --no-interactive (and/or --json) to skip prompts. With --json, stdout is one JSON document and logs go to stderr.

`agent --json` schema

Field	Type	Notes
`schema_version`	`int`	Currently `1`.
`status`	`"ok"` \| `"error"`	Top-level result.
`run_id`	`string` \| `null`	Use with `show` / `engineer` / `run`.
`prompt`	`string`
`url`	`string` \| `null`
`mode`	`string` \| `null`	`"auto"`, `"chrome-mcp"`, or `"agent-browser"`.
`har_path`	`string` \| `null`	Captured HAR.
`script_path`	`string` \| `null`	Generated client.
`usage`	`object`	`{input_tokens, output_tokens, total_cost}`.
`error`	`string` \| `null`	When `status == "error"`.

Exit codes

Code	Meaning
`0`	Success.
`1`	Runtime error.
`2`	Missing required arg under `--no-interactive` / `--json`.

For run, the exit code is the underlying script's return code on success, 1 if no script was found, or non-zero if --no-interactive would have had to prompt.

Output locations

~/.reverse-api/runs/scripts/{run_id}/: permanent storage
./scripts/{descriptive_name}/: local copy with a readable name
Collector: ./collected/{folder_name}/ (items.json, items.csv, README.md)

Caveats

Generated code runs locally via Claude Code, so review before executing.
Sites with aggressive bot detection may block capture or require manual interaction.

Development

git clone https://github.com/kalil0321/reverse-api-engineer.git
cd reverse-api-engineer
uv sync
uv run reverse-api-engineer

Build: ./scripts/clean_build.sh. Requires Python 3.11+ and an API key for agent mode. Manual mode additionally needs the [manual] extra plus playwright install chromium.

License

MIT. See LICENSE.

Frequently asked questions

Is reverse-api-engineer free to use?

reverse-api-engineer is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does reverse-api-engineer do?

The agent that turns websites into APIs!

What is reverse-api-engineer written in?

reverse-api-engineer is primarily written in Python. Its source is publicly available at https://github.com/kalil0321/reverse-api-engineer, and it has 2,454 GitHub stars.