scrapy-playwright is a Scrapy download handler that performs requests using Playwright for Python, so existing Scrapy crawlers can render pages that require JavaScript — built for Python developers who want browser-grade rendering inside a normal Scrapy workflow.

What it is

scrapy-playwright is a download handler plugin for Scrapy, the Python web-scraping framework. Instead of fetching pages through Scrapy's default HTTP machinery, it routes requests to Playwright for Python, Microsoft's browser automation library, which drives real browser engines. It lives in the Scrapy plugin ecosystem, is published under the scrapy-plugins GitHub organisation, is written in Python, and is released under the BSD-3-Clause licence. Distribution is through PyPI, with documentation hosted at scrapy-playwright.readthedocs.io.

The problem it solves is specific. Many modern pages deliver their content through JavaScript, so a plain Scrapy request returns an incomplete document that yields nothing useful to a spider's parsing code. The usual workaround is a separate browser automation script, which means abandoning Scrapy's scheduler, middlewares and item pipelines. scrapy-playwright replaces that split by substituting the download handler rather than the framework, so request scheduling and item processing continue as before.

Key capabilities

• A Scrapy download handler that performs requests using Playwright for Python, replacing the default download path rather than the surrounding framework.
• Renders pages that require JavaScript, so responses arrive at the parsing stage with client-side content already executed.
• Adheres to the regular Scrapy workflow, explicitly without interfering with request scheduling or item processing.
• Drives multiple headless browser engines; the topic list names chrome-headless, firefox-headless and webkit-headless.
• Built for python-asyncio and python3, matching the asynchronous Playwright for Python API.
• Published on PyPI with declared supported Python versions, and documented at scrapy-playwright.readthedocs.io.
• Tested through GitHub Actions with coverage reported by Codecov, and licensed BSD-3-Clause.

Who uses it and how

• Scrapy users who already have spiders, middlewares and item pipelines in place and need JavaScript-rendered pages without rewriting them as standalone Playwright scripts.
• Teams scraping sites that serve content client-side, where Scrapy's default downloader returns a document that is missing the data they need.
• Projects that must target more than one browser engine, since the topics cover Chrome, Firefox and WebKit in headless mode.
• asyncio-based Python services and crawlers, given the python-asyncio and python3 topics and the asynchronous API it wraps.
• Contributors around the scrapy-plugins organisation; the repository carries the hacktoberfest topic alongside 163 forks and 22 open issues, which points to an external contributor base.

Getting started

Install the package from PyPI as scrapy-playwright and enable it as the download handler in a Scrapy project; the setup steps are documented at scrapy-playwright.readthedocs.io.

How it compares

The facts name two adjacent tools: Scrapy itself and Playwright for Python. scrapy-playwright sits between them rather than competing with either, since it does not replace Scrapy's scheduling, middleware or item pipeline, and it does not replace Playwright's browser automation, which it delegates to. For anyone already running Scrapy, it is the integration layer that brings Playwright rendering into the existing framework instead of maintaining two separate stacks.

When to use it — and when not to

A self-hoster takes on operating headless browser engines such as Chromium, Firefox or WebKit alongside the crawler, which is considerably heavier than plain HTTP fetching and is the main operational cost of this route. Projects that only need static HTML should stay with Scrapy's default downloader, where a browser adds nothing. The README excerpt is sparse — largely badges and a short description — so the substantive guidance lives in the Read the Docs site rather than on GitHub, and that documentation should be read before adoption.