Grab is a Python web scraping framework that gives developers helpers to make network requests, scrape websites, and process the scraped content, and it is aimed at Python programmers who build crawlers and data-extraction pipelines.

What it is

Grab is a Python web scraping framework distributed under the MIT licence, with documentation hosted at grab.readthedocs.io. It lives in the Python ecosystem and ships two layers: the core Grab object, which performs network requests and extracts data from the DOM tree of HTML documents with XPath queries, and an asynchronous part called Spider, which is a framework for writing web-site scrapers that run many parallel network requests.

The concrete problem it solves is the glue work that otherwise sits between a low-level HTTP client such as pycurl and separate parsing code. Grab bundles session and cookie handling, HTTP and SOCKS proxy support with or without authorization, Keep-Alive, IDN support, web form tools, multipart file uploading, flexible HTTP request customization, and automatic charset detection into one library, then adds XPath-based extraction on top. For larger crawls, Spider adds rules and conventions that separate request logic from parse logic, automatic processing of network errors where failed tasks return to the task queue, caching of network results in permanent storage, and pluggable task queue backends.

Key capabilities

• Automatic cookies and session support, so a login flow carried out through g.doc.set_input() and g.doc.submit() persists across later requests.
• HTTP and SOCKS proxy support, with or without authorization, plus Keep-Alive and IDN support.
• Web form handling and easy multipart file uploading.
• Flexible customization of HTTP requests and automatic charset detection.
• XPath extraction over the DOM tree, with g.doc.select(), g.doc(...).assert_exists(), .text(), .attr(), and g.make_url_absolute().
• An asynchronous Spider API capable of thousands of simultaneous queries, written as a Spider subclass whose task_generator yields Task objects.
• Task queue backends for Spider: in-memory, redis, and mongodb, together with caching of network results in permanent storage and tools to debug and collect statistics.

Who uses it and how

• Developers writing site scrapers subclass grab.spider.Spider and yield Task objects from task_generator to fan out requests such as search queries across many terms at once.
• Projects that must route traffic through proxies, including authorized HTTP and SOCKS proxies, use the built-in proxy support rather than bolting one on.
• Larger crawls use the redis or mongodb task queue backends instead of the in-memory queue, and rely on automatic network error handling to push failed tasks back onto the queue.
• Repeat crawls of the same pages use Spider's caching of network results in permanent storage to avoid refetching.
• Operators debugging a run use the bundled statistics and debugging tools; support discussion happens in the @grablab English and @grablab_ru Russian Telegram groups.

Getting started

Install with pip install -U grab, and see the platform-specific installation notes at https://grab.readthedocs.io/en/stable/usage/installation.html.

How it compares

No list of paid products that this project replaces is provided in the facts, and no comparable tools are named either, so Grab stands alone in this registry on that axis. Its distinguishing position is the MIT licence combined with a pure-Python library that can be self-hosted inside an existing Python application without a hosted service.

When to use it — and when not to

The maintainer reset the project files to the state of PyPI release 0.6.41 from June 2018 and states plainly that there are no new features: the 2025 work was a compatibility update so the code base runs on Python 2.7 through 3.13, with GitHub Actions running the tests on 2.7 and 3.13. That makes Grab a reasonable choice for teams that want a stable, MIT-licensed scraping library with an existing 2018-era API, and a poor choice for anyone expecting an actively developed feature roadmap. Self-hosters using Spider should expect to operate supporting infrastructure such as a redis or mongodb task queue and a permanent cache store, and anyone upgrading must change imports that previously took DataNotFound or ResponseNotValid from weblib.error, because Grab now raises grab.errors.DataNotFound and InvalidResponseError; pinning grab<1.0 preserves the old behaviour.

import logging

from grab import Grab

logging.basicConfig(level=logging.DEBUG)

g = Grab()

g.go('https://github.com/login')
g.doc.set_input('login', '****')
g.doc.set_input('password', '****')
g.doc.submit()

g.doc.save('/tmp/x.html')

g.doc('//ul[@id="user-links"]//button[contains(@class, "signout")]').assert_exists()

home_url = g.doc('//a[contains(@class, "header-nav-link name")]/@href').text()
repo_url = home_url + '?tab=repositories'

g.go(repo_url)

for elem in g.doc.select('//h3[@class="repo-list-name"]/a'):
    print('%s: %s' % (elem.text(),
                      g.make_url_absolute(elem.attr('href'))))

import logging

from grab.spider import Spider, Task

logging.basicConfig(level=logging.DEBUG)


class ExampleSpider(Spider):
    def task_generator(self):
        for lang in 'python', 'ruby', 'perl':
            url = 'https://www.google.com/search?q=%s' % lang
            yield Task('search', url=url, lang=lang)

    def task_search(self, grab, task):
        print('%s: %s' % (task.lang,
                          grab.doc('//div[@class="s"]//cite').text()))


bot = ExampleSpider(thread_number=2)
bot.run()