Scrapy is a web scraping framework written in Python, distributed under the BSD-3-Clause license, and described in its own README as a tool to extract structured data from websites. It is cross-platform and requires Python 3.10 or newer. The project is maintained by Zyte, formerly known as Scrapinghub, together with a broader group of contributors, and it lives in the Python ecosystem as an installable library rather than a hosted service. Its repository is roughly seventeen years old, which places it among the longer-running projects in the web scraping space.
BSD-3-Clause · Python · Apify vs scrapy →
Crawlee is a web scraping and browser automation library for Node.js, written in TypeScript and published on npm as the `crawlee` package. It lives in the JavaScript and TypeScript ecosystem, and it covers crawling and scraping end to end: fetching pages, following links, extracting data, and storing the results to disk or to the cloud. The project also has a Python counterpart, Crawlee for Python, maintained alongside it by Apify.
Apache-2.0 · TypeScript · Apify vs crawlee →
Crawlee is a web scraping and browser automation library for Python, published on PyPI as the `crawlee` package under the Apache-2.0 licence and developed in the open by Apify. It covers crawling and scraping end to end: crawlers follow links across the web, extract data, and persistently store it in machine-readable formats, while the library handles the technical details around them. It works with Parsel, BeautifulSoup, Playwright, and raw HTTP, runs in both headful and headless mode, and includes proxy rotation. A TypeScript implementation of the same project exists for JavaScript and TypeScript users.
Apache-2.0 · Python · Apify vs crawlee-python →