ScrapeCraft is an MIT-licensed, web-based scraping editor — described in its README as similar to Cursor but specialized for web scraping — that lets users build, test, and deploy scrapers through a visual workflow builder and natural-language AI assistance, and it is aimed at developers and data teams who already work in Python or TypeScript and want extraction pipelines assembled rather than hand-coded.

What it is

ScrapeCraft is a full-stack application, not a library. It pairs a React and TypeScript frontend, styled with Tailwind CSS, against a FastAPI backend that uses LangGraph and the ScrapeGraphAI API to drive scraping work. State lives in PostgreSQL, with Redis as cache, and the whole stack ships as Docker and Docker Compose services. The editor exposes a pipeline model directly: a user creates a pipeline, registers URLs, defines an extraction schema, asks the AI assistant to generate scraping code, runs the pipeline, and exports the results. An AI assistant backed by OpenRouter and the Kimi-k2 model sits inside the editor and is reachable through POST /api/chat/message.

The problem it solves is the assembly work around scraping rather than scraping itself. Instead of writing one-off crawler scripts, wiring ad-hoc parsing, and hand-rolling a viewer for the output, ScrapeCraft treats a scraper as a stored pipeline with a declared schema. Fields are defined with Pydantic, so the shape of the extracted data is explicit before any code is generated, and generated Python carries async support. Pipelines are persisted and addressable: GET /api/pipelines lists them, POST /api/pipelines creates one, PUT /api/pipelines/{id} updates it, and POST /api/pipelines/{id}/run executes it.

Key capabilities

• AI assistant powered by OpenRouter using the Kimi-k2 model, exposed at POST /api/chat/message.
• Multi-URL bulk scraping, managed through the editor's URL Manager.
• Dynamic schema definition with Pydantic for the fields to be extracted.
• Python code generation with async support.
• Real-time streaming over WebSocket at WS /ws/{pipeline_id}.
• Results visualization in table and JSON views, with export to JSON or CSV.
• Watchtower-based auto-updating deployment: push new Docker images to a registry and containers update without manual intervention.

Who uses it and how

• Teams self-hosting the stack with docker compose up -d, reaching the frontend on port 3000 and the API on port 8000, with Swagger-style documentation at /docs.
• Extraction teams that want a schema-first workflow, where Pydantic fields are declared before the AI generates scraping code, so output shape is agreed up front.
• Developers working without Docker: the backend runs under uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 and the frontend under npm start.
• Operators who publish images to their own registry and let Watchtower roll containers forward unattended.
• Users running the same pipeline shape across many URLs at once rather than one scraper per page.

Getting started

Clone the repository, copy .env.example to .env, add an OPENROUTER_API_KEY from OpenRouter and a SCRAPEGRAPH_API_KEY from ScrapeGraphAI, then run docker compose up -d. Development mode avoids Docker entirely and runs the FastAPI backend and the npm frontend separately.

How it compares

No paid product list is supplied, so placement is relative to the tools the README names. ScrapeGraphAI is the scraping engine underneath, LangGraph orchestrates the backend flow, and the Cursor analogy frames the editing experience as AI-assisted authoring rather than configuration. It sits above those libraries as an application layer, not as a replacement for them.

When to use it — and when not to

A self-hoster operates PostgreSQL and Redis, supplies a JWT_SECRET for token signing, and must hold working OpenRouter and ScrapeGraphAI API keys before any pipeline runs, so the tool cannot function as a fully offline stack. Teams that want a managed hosted option, or that are unwilling to depend on two third-party API accounts, are better served elsewhere. The README documents no hosted deployment and no access-control model beyond the JWT secret variable, which makes it a poor fit for multi-tenant or externally exposed installations without additional work.

cd backend
pip install -r requirements.txt
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

cd frontend
npm install
npm start