[opensourcedex]

deepcrawl is a free, open source data extraction & web scraping project written in TypeScript and released under MIT. It has 673 GitHub stars, 79 forks and 2 open issues, and was last pushed 25 days ago. On this registry it ranks #97 of 105 tracked projects in Data Extraction & Web Scraping, with 5 head-to-head comparisons available.

Deepcrawl

WARNING: DO NOT USE DEEPCRAWL IN PRODUCTION RIGHT NOW AS IT IS SUBJECT TO CHANGE AND STILL UNDER RAPID DEVELOPMENT. USE AT YOUR OWN RISK!

100% free and open-source Firecrawl alternative with better performance and flexibility.

Ask DeepWiki about this repo

Ask DeepWiki about this repo

shots

NOTE: DeepCrawl doesn’t target anti-scraping or anti-bot purposes. It’s optimized for high‑frequency agent workloads that scrape public pages to extract cleaned Markdown and a hierarchical links tree.

Deepcrawl is an agent-oriented website data context extraction platform. It extracts cleaned markdown of page content, agent-favoured hierarchical links tree, and metadata that LLMs can digest with minimal token cost to reduce context switching and hallucination.

Full Platform (Nextjs Dashboard, API Workers, Auth Workers, and Database) is open and transparent.

Documentation

Visit https://deepcrawl.dev/docs to view the documentation.

Contributing

Please read the contributing guide.

License

MIT License

Open Source. Open Code - built with ❤️ by @felixLu.

Star History

Star History Chart

Frequently asked questions

Is deepcrawl free to use?

deepcrawl is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does deepcrawl do?

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.

What is deepcrawl written in?

deepcrawl is primarily written in TypeScript. Its source is publicly available at https://github.com/lumpinif/deepcrawl, and it has 673 GitHub stars.