toapi is a Python library that turns any web page into a JSON API by declaring the fields to extract with CSS selectors, aimed at developers who need structured data from sites that never published an API.

What it is

toapi is a small Python framework, built on Flask, that maps clean API routes onto messy source URLs on public websites. A developer decorates an Item class with @api.site to name the target site, @api.list to point at the repeating elements, and @api.route to declare the outward-facing path, then pulls individual fields out of the markup using the Text and Attr helpers from the htmlparsing package. Running the module starts a Flask server, and each incoming request is fetched, parsed, and returned as JSON. Pages are retrieved with requests, or with a headless browser when one is configured through Api(browser="/path/to/geckodriver"), which covers sites whose content only appears after client-side JavaScript runs.

The concrete problem it solves is the absence of a data interface on ordinary websites. It replaces the hand-written scraper plus custom glue code that a team would otherwise maintain: the crawler loop, the retry handling, the HTML parsing, the caching layer, and the web endpoint that serves the result. Instead of those moving parts, the project offers a declarative description of the data wanted, and it also removes the need for an intermediate database, since pages are fetched and parsed on demand and held in an in-memory cache. Everything sits inside the Python ecosystem, and the HTTP surface it exposes is plain JSON over Flask.

Key capabilities

• Declarative extraction: an Item subclass states the desired fields, and each field is bound to a CSS selector through Text(".titleline > a") or Attr(".titleline > a", "href").
• Route mapping: @api.route("/posts", "/news") and @api.route("/posts?page={page}", "/news?p={page}") translate tidy API paths into source URLs, with {param} placeholders passed through from the request.
• Multi-site aggregation: several websites can be merged behind a single API surface.
• Cleaning hooks: a method named with the clean_ prefix, such as clean_next_page, post-processes a value before it is returned.
• Automatic caching: fetched pages and parsed results are cached in memory, so repeat calls do not re-fetch the source.
• Headless browser support: passing Api(browser="/path/to/geckodriver") routes fetching through a real browser for JavaScript-heavy pages.
• Flask-based serving: api.run(host="127.0.0.1", port=5000) exposes the whole thing as an HTTP endpoint that returns JSON.

Who uses it and how

• Data teams that need a stable JSON endpoint in front of a site which offers none, without writing or babysitting a crawler.
• Prototypers who want a working API over a page in a few dozen lines, as the Hacker News example in the README shows with a Post item returning titles and URLs.
• Teams combining several sources, using the multi-site support to present one API across multiple websites.
• Developers facing JavaScript-rendered pages, who configure the headless browser option rather than reverse-engineering the underlying requests.
• Projects that want to reshape pagination, as in the clean_next_page example that rewrites a morelink href into the local /posts? form.

Getting started

Install with pip install toapi, which requires Python 3.10 or newer, then define an Api, an Item, and its routes in a module and start it with python app.py. The repository is developed with uv sync, uv run pytest, and uv run ruff check ., and ships a CI workflow plus pre-commit hooks.

How it compares

No comparable or paid products are listed in the facts for this entry, so no like-for-like contrast can be drawn here. Among similar tools, it stands alone in this registry.

When to use it — and when not to

A self-hoster runs only a Python process, since the caching is in memory and no database or storage layer is required, but that also means cached results vanish on restart and nothing is shared between instances. It suits small to medium read-mostly endpoints; anyone needing durable storage, scheduled crawling, or high write volume should look elsewhere. The obvious fragility is that field definitions are pinned to the current markup of a third-party site, so a redesign on the source side breaks the selectors, and JavaScript-dependent pages require an external geckodriver installation to be managed alongside the application.

pip install toapi

from htmlparsing import Attr, Text
from toapi import Api, Item

api = Api()


@api.site("https://news.ycombinator.com")
@api.list(".athing")
@api.route("/posts", "/news")
@api.route("/posts?page={page}", "/news?p={page}")
class Post(Item):
    title = Text(".titleline > a")
    url = Attr(".titleline > a", "href")


api.run(host="127.0.0.1", port=5000)

python app.py

{
  "Post": [
    {"title": "Mathematicians Crack the Cursed Curve", "url": "https://www.quantamagazine.org/..."},
    {"title": "Stuffing a Tesla Drivetrain into a 1981 Honda Accord", "url": "https://jalopnik.com/..."}
  ]
}

   ┌────────────┐    ┌────────────┐    ┌────────────┐
   │  /posts    │ ─▶ │  fetch     │ ─▶ │  parse     │ ─▶  JSON
   │  (route)   │    │  (cache)   │    │  (Item)    │
   └────────────┘    └────────────┘    └────────────┘

@api.site("https://news.ycombinator.com")
@api.route("/posts", "/news")
class Page(Item):
    next_page = Attr(".morelink", "href")

    def clean_next_page(self, value):
        return f"/posts?{value.split('?', 1)[1]}"

git clone https://github.com/elliotgao2/toapi.git
cd toapi
uv sync          # install deps into .venv
uv run pytest    # run tests
uv run ruff check .

uv run pre-commit install