Selectolax is a Python binding to the Modest and Lexbor C engines, exposed through Cython, that gives Python developers a fast HTML5 parser with CSS selector queries for scraping, crawling, and data extraction work.

What it is

Selectolax is a Python library that binds to two C HTML5 parsing engines, Modest and Lexbor, and wraps them with Cython so that parsing and selector matching run in native code while the public interface stays ordinary Python. Parsers are constructed directly from an HTML string, for example LexborHTMLParser(html), and the resulting tree exposes nodes that can be queried and read through small, consistent methods. Both engines are available as backends, and their feature sets are described as almost identical.

The concrete problem it solves is the cost of parsing and querying HTML in pure Python. Selectolax lives in the Python web-scraping ecosystem and is positioned there as an alternative to Beautiful Soup, which is the package named as the comparison point in the README's benchmark table. The project's own benchmark extracts a title, links, scripts, and a meta tag from the main pages of the top 754 domains, and the script for it ships at examples/benchmark.py.

Key capabilities

• Two parsing backends, Modest and Lexbor, both exposed to Python. Lexbor is the preferred backend as of 2024, while Modest remains available for compatibility reasons even though its underlying C library is no longer maintained.
• CSS selector querying through css() and css_first(), which return node collections and a single node respectively, as in tree.css_first('h1#title').text().
• Node inspection covering text, attributes, tag name, parent tag, and serialized HTML, read via node.text(), node.attributes, node.tag, node.parent.tag, and node.html.
• Advanced CSS selector support, including structural and relational forms such as div > :nth-child(2n+1):not(:has(a)).
• A lexbor-contains CSS pseudo-class for matching text, supporting a case-insensitive mode through the i flag, as in p:lexbor-contains("awesome" i).
• Installation from PyPI with pip install selectolax, plus a selectolax[cython] extra for cases where compilation fails.
• A development build path from source using git clone --recursive, make clean, and make dev.

Who uses it and how

• Scrapers and crawlers, as reflected in the project's topics covering crawler, scraper, and web-scraping use cases.
• Extraction pipelines that read text out of many documents per run, such as the title, links, scripts, and meta tag pulled across the top 754 domains in the bundled benchmark.
• Data extraction and analytics work, which is the registry category the project is filed under.
• Teams that need precise structural selection, using selectors such as :nth-child and :has rather than post-processing results in Python.
• Workflows that need text matching inside the selector itself, using lexbor-contains with or without the case-insensitive flag.

Getting started

Install from PyPI with pip install selectolax; if installation fails on compilation errors, use pip install selectolax[cython]. The development version is built from a recursive clone of the repository followed by pip install -r requirements_dev.txt and python setup.py install.

How it compares

Among similar tools named in the available facts, Beautiful Soup is the direct comparison point, appearing in the README benchmark that measures extraction of a title, links, scripts, and a meta tag across the top 754 domains. No timing figures beyond the package names are captured in the facts provided, so the benchmark table itself is the place to look for the actual numbers. The two backends also compare against each other internally, with Lexbor recommended and Modest kept only for compatibility.

When to use it — and when not to

Choose it when CSS selector queries over HTML5 are the core of the job and the environment can install a Cython extension at build or wheel time. Avoid it if the runtime has no compiler and no suitable wheel for the target Python version, since older releases can fail to build against newer Python, and avoid relying on the Modest backend for new work because its underlying C library is unmaintained. The README covers installation, backend choice, and a handful of examples, but leaves deployment, concurrency, and memory behaviour undocumented, so those aspects need testing before committing to the library at scale.

pip install selectolax

pip install selectolax[cython]

git clone --recursive  https://github.com/rushter/selectolax
cd selectolax
pip install -r requirements_dev.txt
python setup.py install

make clean
make dev

In [1]: from selectolax.lexbor import LexborHTMLParser
   ...:
   ...: html = """
   ...: <h1 id="title" data-updated="20201101">Hi there</h1>
   ...: <div class="post">Lorem Ipsum is simply dummy text of the printing and typesetting industry. </div>
   ...: <div class="post">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</div>
   ...: """
   ...: tree = LexborHTMLParser(html)

In [2]: tree.css_first('h1#title').text()
Out[2]: 'Hi there'

In [3]: tree.css_first('h1#title').attributes
Out[3]: {'id': 'title', 'data-updated': '20201101'}

In [4]: [node.text() for node in tree.css('.post')]
Out[4]:
['Lorem Ipsum is simply dummy text of the printing and typesetting industry. ',
 'Lorem ipsum dolor sit amet, consectetur adipiscing elit.']

In [1]: html = "<div><p id=p1><p id=p2><p id=p3><a>link</a><p id=p4><p id=p5>text<p id=p6></div>"
   ...: selector = "div > :nth-child(2n+1):not(:has(a))"

In [2]: for node in LexborHTMLParser(html).css(selector):
   ...:     print(node.attributes, node.text(), node.tag)
   ...:     print(node.parent.tag)
   ...:     print(node.html)
   ...:
{'id': 'p1'}  p
div
<p id="p1"></p>
{'id': 'p5'} text p
div
<p id="p5">text</p>

from selectolax.lexbor import LexborHTMLParser
html = "<div><p>hello </p><p id='main'>lexbor is AwesOme</p></div>"
parser = LexborHTMLParser(html)
# Case-insensitive search
results = parser.css('p:lexbor-contains("awesome" i)')
# Case-sensitive search
results = parser.css('p:lexbor-contains("AwesOme")')
assert len(results) == 1
assert results[0].text() == "lexbor is AwesOme"

In [1]: from selectolax.lexbor import LexborHTMLParser

In [2]: html = """
   ...: <title>Hi there</title>
   ...: <div id="updated">2021-08-15</div>
   ...: """

In [3]: parser = LexborHTMLParser(html)
In [4]: parser.root.css_first("#updated").text()
Out[4]: '2021-08-15'