gain is an MIT-licensed, asyncio-based web crawling framework for Python that lets developers declare items and parsers while the framework handles concurrency, retries, and persistence on their behalf.

What it is

gain is a web crawling framework written in Python and published on PyPI as the gain package. It is built on asyncio and aiohttp for asynchronous network transport and on lxml/pyquery for HTML parsing, with an optional uvloop event loop on Linux. It lives in the Python data-extraction ecosystem and targets developers who want spiders as small Python programs rather than a large externally configured crawler. The repository last received a push on 2026-08-09.

What it replaces is the glue code a developer would otherwise hand-write around aiohttp: the URL queue, the concurrency budget, retry handling, link-following rules, and the scheduling of parse and save steps. A spider declares start_url, concurrency, headers, and a list of Parser objects; each parser is either a follow rule matched against URLs by regular expression, or an extract rule that instantiates an Item from matching pages. Extraction uses Css, XPathParser, or Regex selectors, and results reach an async save() hook the developer defines. That loop, from start_url through parser and item to persistence, is the whole model.

Key capabilities

• Declarative Spider class carrying start_url, concurrency, headers, and parsers, started by calling MySpider.run().
• Regex-driven Parser objects that either follow links (one argument) or extract into an Item (two arguments).
• Field selectors Css, XPathParser, and Regex for pulling values out of HTML.
• Async save() method on Item as the persistence hook, demonstrated writing to a file with aiofiles.
• Per-spider proxy attribute for routing requests through a proxy.
• Linux uvloop extra installed with pip install "gain[uvloop]".
• Runnable scripts in the example/ directory targeting Scrapinghub, V2EX, and Sciencenet.

Who uses it and how

• Individual developers writing a single spider file and running it with python spider.py against a blog or listing site.
• Teams needing custom storage, because save() can write a file, push to a queue, or insert into a database instead of relying on a built-in backend.
• Scrapers that must route traffic through a proxy, set via proxy alongside a bounded concurrency such as 5.
• Developers crawling paginated archives, using one follow parser for page URLs and a second extract parser for the article pattern.
• Projects on Python 3.10 or newer, the minimum supported interpreter.

Getting started

Install from PyPI with pip install gain and run a spider file with python spider.py; on Linux, pip install "gain[uvloop]" adds the faster event loop. Contributors clone the repository and use uv sync, uv run pytest, and uv run ruff check ..

How it compares

The facts provided name no paid products that gain replaces, and name no comparable crawling tools. On the evidence available, it stands alone in this registry, so no side-by-side comparison of licence, self-hosting, data ownership, or cost model can be made.

When to use it — and when not to

Choose it when a small, readable async spider with a declarative item and parser model is enough and the project already runs Python 3.10 or newer. A self-hoster must supply storage, scheduling, and deduplication: save() is an empty hook, and no database, object store, or job queue ships with the framework. Anyone who needs a managed hosted scraping service, a non-Python stack, or a batteries-included pipeline should look elsewhere, and because the README is brief, prospective users should read the example/ scripts before committing.

pip install gain

pip install "gain[uvloop]"

import aiofiles
from gain import Css, Item, Parser, Spider


class Post(Item):
    title = Css(".entry-title")
    content = Css(".entry-content")

    async def save(self):
        async with aiofiles.open("scrapinghub.txt", "a+") as f:
            await f.write(self.results["title"] + "\n")


class MySpider(Spider):
    concurrency = 5
    headers = {"User-Agent": "Google Spider"}
    start_url = "https://blog.scrapinghub.com/"
    parsers = [
        Parser(r"https://blog.scrapinghub.com/page/\d+/"),
        Parser(r"https://blog.scrapinghub.com/\d{4}/\d{2}/\d{2}/[a-z0-9\-]+/", Post),
    ]


MySpider.run()

python spider.py

from gain import Css, Item, Parser, Spider, XPathParser


class Post(Item):
    title = Css(".breadcrumb_last")

    async def save(self):
        print(self.title)


class MySpider(Spider):
    start_url = "https://mydramatime.com/europe-and-us-drama/"
    concurrency = 5
    headers = {"User-Agent": "Google Spider"}
    parsers = [
        XPathParser('//span[@class="category-name"]/a/@href'),
        XPathParser('//div[contains(@class, "pagination")]/ul/li/a[contains(@href, "page")]/@href'),
        XPathParser('//div[@class="mini-left"]//div[contains(@class, "mini-title")]/a/@href', Post),
    ]
    proxy = "https://localhost:1234"


MySpider.run()

   ┌────────────┐    ┌────────────┐    ┌────────────┐    ┌────────────┐
   │  start_url │ ─▶ │  Parser    │ ─▶ │  Item      │ ─▶ │ save()     │
   │            │    │  (follow)  │    │  (extract) │    │  (persist) │
   └────────────┘    └────────────┘    └────────────┘    └────────────┘
                          ▲                                      │
                          └──────────── new urls ────────────────┘

git clone https://github.com/elliotgao2/gain.git
cd gain
uv sync                 # install deps into .venv
uv run pytest           # run tests
uv run ruff check .     # lint

uv run pre-commit install