Scrapling is an adaptive Python web scraping framework for developers, data teams, and regular users who need anything from a single request to a full-scale crawl without rewriting selectors every time a site changes.

What it is

Scrapling is an open-source Python framework for web scraping that spans the whole range from one fetch to a concurrent crawl. Its parser learns from website changes and automatically relocates your elements when pages update, so a selector that worked before a redesign can still find the same data afterward. Its fetchers are built to bypass anti-bot systems such as Cloudflare Turnstile out of the box, and its spider framework scales to concurrent, multi-session crawls with pause and resume, automatic proxy rotation, and a crawl speed that adapts to how fast each website responds and backs off when the site starts blocking.

The problem it solves is the maintenance loop that follows any scraping project: hand-written CSS and XPath selectors break on layout changes, and the surrounding plumbing for headless browsers, proxy rotation, retries, and pacing is usually written from scratch. Scrapling assembles those pieces into one library in the Python ecosystem, exposing a Spider API modelled on the familiar Scrapy pattern alongside its own fetchers and an MCP server for agent workflows.

Key capabilities

• Four fetchers cover different targets: Fetcher, AsyncFetcher, StealthyFetcher, and DynamicFetcher, with options such as headless=True and network_idle=True on StealthyFetcher.fetch.
• Adaptive element tracking works through p.css('.product', auto_save=True) at first scrape and adaptive=True later, after the website structure changes.
• The spider framework is imported as from scrapling.spiders import Spider, Response, and a crawl is defined with name, start_urls, and an async def parse method, then started with MySpider().start().
• Crawls run concurrently across multiple sessions with pause and resume, automatic proxy rotation, real-time stats, and streaming output.
• Crawl speed adapts to each website's response speed and backs off automatically when blocking begins.
• Cloudflare Turnstile is handled out of the box, without additional services.
• A CLI and an MCP server are included, matching the project's mcp, mcp-server, and cli topics.

Who uses it and how

• Individual users who need a single request or a small scrape and want session handling and anti-bot bypass without assembling their own stack.
• Data teams running full crawls that must pause, resume, and rotate proxies across long jobs.
• AI and agent builders who connect to the bundled MCP server, indicated by the project's ai-scraping, mcp, and mcp-server topics.
• Scraping operations that pair the framework with residential or datacenter proxy providers, several of which sponsor the project.
• A large community around the project, with 81,427 stars, 8,254 forks, and a Discord server for support and discussion.

Getting started

The library installs and imports as scrapling, as shown by from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher. Full installation and deployment instructions are published at https://scrapling.readthedocs.io/en/latest/.

How it compares

Within the facts, the nearest named reference points are Scrapy, whose spider API Scrapling echoes, and Playwright, which appears among the project's topics and which Scrapling's browser-based fetchers cover in spirit. Where Scrapy-style projects expect you to bring your own adaptive selection and anti-bot handling, Scrapling ships those behaviours as part of the same library.

When to use it — and when not to

A self-hoster still operates a Python runtime and headless browser sessions, and at crawl scale also supplies proxies, so the framework is not a zero-infrastructure option. Sites behind enterprise-grade protection such as Akamai, DataDome, Kasada, or Incapsula are outside what the framework handles by default, since the README points to third-party API endpoints that generate valid anti-bot tokens for those. Teams that only need static HTML parsing and no browser automation or anti-bot bypass will find a lighter library sufficient.

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)  # Fetch website under the radar!
products = p.css('.product', auto_save=True)                                        # Scrape data that survives website design changes!
products = p.css('.product', adaptive=True)                                         # Later, if the website structure changes, pass `adaptive=True` to find them!

from scrapling.spiders import Spider, Response

class MySpider(Spider):
  name = "demo"
  start_urls = ["https://example.com/"]

  async def parse(self, response: Response):
      for item in response.css('.product'):
          yield {"title": item.css('h2::text').get()}

MySpider().start()