What it is
Scrapy is a web scraping framework written in Python, distributed under the BSD-3-Clause license, and described in its own README as a tool to extract structured data from websites. It is cross-platform and requires Python 3.10 or newer. The project is maintained by Zyte, formerly known as Scrapinghub, together with a broader group of contributors, and it lives in the Python ecosystem as an installable library rather than a hosted service. Its repository is roughly seventeen years old, which places it among the longer-running projects in the web scraping space.
The concrete problem it solves is the gap between a one-off script that fetches a page and a repeatable system that pulls structured data out of many pages. Developers who need to crawl sites and turn the results into structured records would otherwise assemble their own fetching, parsing, and scheduling code. Scrapy packages that work as a framework, so the crawling and scraping logic is written against an existing structure instead of being rebuilt for each project. The topic list on the repository — crawler, crawling, framework, scraping, web-scraping, web-scraping-python — reflects that framing directly.
Key capabilities
- Crawls websites and extracts structured data from the pages it retrieves, as stated in the project README.
- Ships as a framework, so scraping projects are built on top of it rather than written from scratch.
- Runs cross-platform, without a platform-specific dependency noted in the README.
- Requires Python 3.10 or newer, and supports the Python versions listed on its PyPI badge.
- Publishes releases to PyPI, where the current version is tracked by the project's version badge.
- Publishes a conda-forge build, tracked by a separate conda version badge.
- Maintains test and coverage badges, indicating continuous integration and coverage reporting on the master branch.
Who uses it and how
- Python developers who need to extract structured data from websites and want a framework rather than a bespoke script.
- Teams running crawls across platforms, since the README states the project is cross-platform.
- Projects on Python 3.10 or newer, which is the minimum the README specifies.
- Users installing from PyPI with pip, or from conda-forge where a package is published.
- Contributors, including participants in Hacktoberfest, which appears in the repository topic list.
Getting started
Install with pip install scrapy, or use the conda-forge package. The README then points to the documentation at docs.scrapy.org for
project readme (upstream, from github) — read inline
|logo|
.. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg
:target: https://scrapy.org
:alt: Scrapy
:width: 480px
|version| |python_version| |tests| |coverage| |conda| |deepwiki|
.. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: PyPI Version
.. |python_version| image:: https://img.shields.io/pypi/pyversions/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: Supported Python Versions
.. |tests| image:: https://img.shields.io/github/check-runs/scrapy/scrapy/master?label=tests
:target: https://github.com/scrapy/scrapy/actions?query=branch%3Amaster
:alt: Tests
.. |coverage| image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg
:target: https://codecov.io/github/scrapy/scrapy?branch=master
:alt: Coverage report
.. |conda| image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg
:target: https://anaconda.org/conda-forge/scrapy
:alt: Conda Version
.. |deepwiki| image:: https://deepwiki.com/badge.svg
:target: https://deepwiki.com/scrapy/scrapy
:alt: Ask DeepWiki
Scrapy_ is a web scraping framework to extract structured data from websites.
It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_
(formerly Scrapinghub) and many other contributors_.
.. _many other contributors: https://github.com/scrapy/scrapy/graphs/contributors
.. _Scrapy: https://scrapy.org/
.. _Zyte: https://www.zyte.com/
Install with:
.. code:: bash
pip install scrapy
And follow the documentation_ to learn how to use it.
.. _documentation: https://docs.scrapy.org/en/latest/
If you wish to contribute, see Contributing_.
.. _Contributing: https://docs.scrapy.org/en/master/contributing.html
