Deepcrawl is a free, MIT-licensed, agent-oriented website data extraction platform that turns public pages into cleaned Markdown, a hierarchical links tree and LLM-ready metadata, built for developers running AI agents who want to self-host the whole stack on Cloudflare or Vercel rather than pay for Firecrawl.
What it is
Deepcrawl is a TypeScript platform that extracts website data in shapes an AI agent can actually consume. It lives in the edge and JavaScript ecosystem: the codebase combines Next.js 16, Cloudflare Workers, Hono and oRPC, with Better Auth handling identity and the AI SDK wiring extraction output into agent workflows. The project describes itself as 100% free and fully open source, and it publishes the full platform — Next.js dashboard, API workers, auth workers and database — rather than keeping part of the stack proprietary. It ships under the MIT licence and is deployable to Cloudflare or Vercel by the person running it.
The concrete problem it solves is context cost. Feeding raw HTML to a model burns tokens, invites context switching and increases hallucination. Deepcrawl instead returns three things: cleaned Markdown of page content, an agent-favoured hierarchical links tree, and metadata that LLMs can digest at minimal token cost. It is positioned explicitly as an alternative to Firecrawl, aiming for better performance and flexibility. One boundary matters: Deepcrawl does not target anti-scraping or anti-bot work. It is optimised for high-frequency agent workloads that scrape public pages.
Key capabilities
- Cleaned Markdown extraction of page content, produced through the project's HTML cleaner and HTML-to-Markdown pipeline.
- Hierarchical links tree extraction (
links-tree, links-extraction topics), which is the capability the project claims as its edge over alternatives.
- Metadata extraction designed so LLMs digest pages with minimal token cost, reducing context switching and hallucination.
- Edge-deployable API workers built on Cloudflare Workers, with Hono as the web framework and oRPC for typed API contracts.
- Auth workers using Better Auth, so the self-hosted deployment carries its own identity layer.
- Next.js 16 dashboard that administers the crawling service, alongside the API and auth workers.
- AI SDK and agent tooling integration, so extraction output can be wired directly into agent runs.
Who uses it and how
- Agent developers who need high-frequency scraping of public pages and want extracted context rather than raw HTML in the prompt.
- Teams that want the data pipeline inside their own cloud account, deploying the workers to Cloudflare or the dashboard to Vercel.
- Operators who need an admin surface: the Next.js dashboard, auth workers and database are part of the same open stack.
- Projects replacing a hosted extraction API on cost grounds, since the licence is MIT and no usage fee attaches to the software.
- Builders whose targets are public documentation, marketing sites and similar public pages, which is the workload Deepcrawl is tuned for.
Getting started
The README points to https://deepcrawl.dev/docs for setup instructions, and contributing is documented in CONTRIBUTING.md. Deployment is self-managed: the platform — Next.js dashboard, API workers, auth workers and database — is deployed to Cloudflare or Vercel by the operator.
How it compares
The named comparison is Firecrawl, which Deepcrawl positions against as a free, full open-source alternative with better links extraction and, per the README, better performance and flexibility. The difference on the axes the project states is cost model and control: MIT-licensed code that the operator deploys and owns, rather than a hosted service. That also shifts the operational burden onto the self-hoster, which is the trade the licence buys.
When to use it — and when not to
A self-hoster operates the full stack: the Next.js dashboard, API workers, auth workers and a database, plus whatever infrastructure Cloudflare or Vercel requires. Do not pick Deepcrawl for production right now — the README carries an explicit warning that it is subject to change, under rapid development, and used at the reader's own risk. It is also the wrong tool for anti-bot or anti-scraping targets, since that is deliberately outside its scope. The repository shows 673 stars, 79 forks and 2 open issues, with the latest push in August 2026.
