proxy_pool is a self-hosted Python proxy IP pool for web spiders that continuously collects, checks, and serves rotating HTTP proxies from public free proxy sources through a Redis-backed pool.

What it is

proxy_pool (爬虫代理IP池) is an open-source proxy pool built for crawler and spider workloads in the Python ecosystem. It runs as a set of processes that scrape free proxy lists from a fixed set of provider sites, verify which of those proxies actually work, and store the survivors so that a scraper can pull a fresh, validated proxy on demand instead of hard-coding one. The pool is backed by Redis, and the project is distributed under the MIT licence.

The concrete problem it replaces is the hand-rolled glue that every spider project otherwise writes: a script that scrapes a proxy list page, a loop that tests each address, and a fragile retry helper inside the crawler itself. proxy_pool centralises that into a reusable service. A crawler asks the pool for a proxy, uses it, and calls delete_proxy(proxy) when the request fails, so bad addresses leave rotation automatically. Because the pool owns verification and eviction, individual spiders no longer need their own proxy logic.

Key capabilities

• Collects free proxies from built-in sources, each implemented as a module under fetcher/sources/ such as kxdaili.py, kuaidaili.py, ip3366.py, ihuan.py, ip89.py, docip.py, goodips.py, daili66.py, proxifly.py, freevpnnode.py, geonode.py, and roundproxies.py.
• Supports custom proxy sources by adding a .py file that inherits the BaseFetcher base class, declares name, url, and enabled, and implements fetch() as a generator that yields proxies in host:port format.
• Auto-discovers new fetchers: the schedule process rescans the fetcher/sources/ directory on its next run, so no configuration edit is required to enable a newly added source.
• Lists active sources with the python proxyPool.py fetcher command.
• Temporarily disables any source by adding its name to the PROXY_FETCHER_EXCLUDE blacklist in setting.py.
• Serves rotating proxies to crawlers over HTTP, with usage shown as requests.get('http://www.example.com', proxies={"http": "http://{}".format(proxy)}).
• Removes failed proxies from the pool via delete_proxy(proxy), keeping rotation clean without manual pruning.

Who uses it and how

• Python crawling teams that already run Redis and want a shared proxy layer instead of per-project proxy scraping code.
• Scrapers that need a retry pattern: fetch a proxy, attempt the request, and call delete_proxy(proxy) on failure so the address is dropped.
• Teams behind long-running collection jobs that need a background schedule process refreshing the pool while an API process serves live requests.
• Operations that need to add private or paid proxy feeds, done by writing a BaseFetcher subclass rather than patching core code.
• Projects seeking a general-purpose pool architecture — the maintainers state the project accepts only generic proxy-pool functionality and does not take on specialised features.

Getting started

Run the project's own Python entry point, python proxyPool.py, with a Redis instance available; use python proxyPool.py fetcher to confirm which proxy sources are enabled before relying on the pool.

How it compares

No comparable tools are named in the registry facts for this entry, so proxy_pool stands alone in this registry rather than being presented alongside alternatives.

When to use it — and when not to

A self-hoster must operate Redis plus the fetcher, schedule, and API processes, so it suits teams that already run that infrastructure. The README is explicit that the built-in sources are free proxies with limited quality, so results may be poor unless a better source is added through BaseFetcher; anyone needing guaranteed, high-availability commercial proxies should look elsewhere. The project also carries 298 open issues and the maintainers position it as a basic general-purpose architecture rather than a feature-complete product, so it is best treated as a foundation to extend.

______                        ______             _
| ___ \_                      | ___ \           | |
| |_/ / \__ __   __  _ __   _ | |_/ /___   ___  | |
|  __/|  _// _ \ \ \/ /| | | ||  __// _ \ / _ \ | |
| |   | | | (_) | >  <img src="https://api.hellogithub.com/v1/widgets/recommend.svg?rid=92a066e658d147cc8bd8397a1cb88183&claim_uid=DR60NequsjP54Lc" alt="Featured｜HelloGitHub"  width="250" height="54" /></a>