Ruia is an asynchronous web scraping micro-framework for Python 3.6 and above, built on asyncio and aiohttp, and intended for Python developers who want to declare what a crawler should collect and let the framework handle concurrent requests.
What it is
Ruia is a small, Apache-2.0 licensed Python library for writing web crawlers, developed under the python-ruia organisation and published to PyPI as the ruia package. Its programming model is declarative: a crawler is assembled from items that describe the data to be collected, a spider that controls the crawl, and request and response objects that carry pages through the run. The documentation is organised around exactly those pieces, with tutorials for defining data items, spider control, requests and responses, custom middleware and plugins. It describes itself as a micro-framework, which is accurate in the sense that it does not attempt to be a full crawling platform — it supplies the async crawling core and leaves storage, scheduling policy and deployment to the developer.
The problem Ruia addresses is the boilerplate that appears when a crawler is built directly on asyncio and aiohttp. Wiring the event loop, dispatching concurrent requests, handling responses, and structuring parsed output by hand is repetitive, and the same shape of code is rewritten for every site. Ruia replaces that hand-rolled plumbing with a declarative spider abstraction sitting on the same underlying stack, so the crawler definition stays short while execution remains asynchronous. It lives squarely in the Python async ecosystem: aiohttp provides the HTTP layer, uvloop is an optional performance extra, and the project credits demiurge as an influence on its design.
Key capabilities
- Declarative spider programming, with separate tutorials for defining data items, spider control, and request and response handling.
- Asynchronous execution powered by
asyncio and aiohttp, distributed as the ruia package for Python 3.6 and later.
- An optional
uvloop extra (pip install -U ruia[uvloop]) for Linux and macOS, with a plain install for Windows.
- Extensibility through middleware and plugins, including a dedicated tutorial on writing plugins and the
awesome-ruia plugin collection.
- JavaScript support, listed among the framework's headline features.
- Debugging and request-limit helpers in the wider organisation:
ruia-cache for caching during debugging to reduce request limits, and ruia-shell for debugging scripts interactively.
- A
black-formatted codebase with an open contribution path for bug reports, plugins, documentation and test cases.
Who uses it and how
- Python developers who need to crawl many URLs concurrently and prefer a declarative spider definition over writing
asyncio and aiohttp code directly.
- Linux and macOS users who install the
uvloop extra to pair the framework with a faster event loop, and Windows users who install the base package.
- Teams that need crawler-specific behaviour in the request/response pipeline, who implement custom middleware or publish plugins through
awesome-ruia.
- Developers debugging and iterating on crawlers, who use
ruia-shell to debug scripts and ruia-cache to cache responses and stay under request limits.
- Scrapers that need to render or handle pages requiring JavaScript support.
Getting started
Install from PyPI with pip install -U ruia[uvloop] on Linux and macOS, or pip install -U ruia on Windows. New features can be installed ahead of release from source with pip install git+https://github.com/howie6879/ruia.
How it compares
Ruia sits among the Python async tooling it is built on: aiohttp supplies the HTTP client, uvloop is an optional accelerator, and the README credits demiurge as an acknowledged influence on the framework. Its distinguishing choice is the micro-framework scope — a small declarative layer over asyncio rather than a broader crawling platform — with extensibility pushed out to middleware and the awesome-ruia plugin ecosystem.
When to use it — and when not to
Because Ruia is a library rather than a hosted service, there is no database, object store or mail server to operate; what a user runs is a Python environment, and scheduling, persistence and proxy management remain the user's responsibility. Anyone who needs distributed crawling across multiple machines should wait: distributed crawling or scraping is still an unchecked item on the project's TODO list. The README is also brief and defers most explanation to external documentation, and the project describes itself as still under development, so users who require a heavily documented or fully managed scraping platform may be better served elsewhere.
