[opensourcedex]

twscrape is a free, open source data extraction & web scraping project written in Python and released under MIT. It has 2,915 GitHub stars, 333 forks and 79 open issues, and was last pushed 3 days ago. On this registry it ranks #64 of 149 tracked projects in Data Extraction & Web Scraping, with 5 head-to-head comparisons available. It gained 100 stars over the last 30 tracked days.

What is twscrape?

twscrape is an MIT-licensed async Python library and CLI that reads X/Twitter Search and GraphQL endpoints through a pool of authorized accounts, built for developers, researchers, and data teams that need tweet and user data in Python without hand-rolling browser automation.

What it is

twscrape is published on PyPI and lives in the Python data-extraction ecosystem, under the Data & Analytics / Data Extraction & Web Scraping category. It exposes X/Twitter Search and GraphQL API methods through an async/await interface, keeps account sessions in SQLite, and rotates accounts automatically when an endpoint is rate-limited. Results come back either as parsed SNScrape-style models or as raw API responses, so downstream code can choose typed objects or the original wrapper.

The problem it solves is account and rate-limit management. Scraping X/Twitter with a single account stalls the moment that account hits a limit, and hand-written rotation code has to track sessions, locks, proxies, and wait timing. twscrape centralises all of that: accounts are added from browser cookies containing auth_token and ct0, each session is stored in an account database, and the pool switches accounts across rate-limited operations on the caller's behalf. Every parsed method also has a _raw counterpart, such as api.search_raw(...), for callers that need the untouched response.

Key capabilities

  • Async/await API for running multiple scrapers concurrently, with gather() for collecting results into a list or async for streaming directly from api.search(...).
  • Account pool built from browser cookies (auth_token and ct0) via twscrape add_cookie my_account; re-running the command replaces the saved session while preserving credentials, statistics, locks, and proxy settings.
  • Session persistence in SQLite, using accounts.db by default and overridable through API("accounts.db").
  • Automatic account switching across rate-limited operations, with raise_when_no_account, wait_timeout, and wait_interval controlling behaviour when no account is free, and NoAccountError raised instead of ending an operation.
  • Parsed SNScrape-compatible models plus _raw variants that return the original Twitter API response wrapper.
  • Per-account proxies, an httpx default HTTP backend, and an optional curl-cffi backend for browser-like TLS fingerprinting through pip install "twscrape[curl]" and TWS_HTTP_BACKEND=curl.
  • Search product selection, defaulting to the Latest tab and switching with kv={"product": "Top"} or kv={"product": "Media"}, alongside a login flow with optional email verification code retrieval.

Who uses it and how

  • Social network analysis and data-collection researchers pulling tweets with CLI queries such as "from:xdevelopers lang:en" at a controlled --limit.
  • Teams running their own account pool for concurrent scraping, where the pool absorbs per-account limits instead of failing the job.
  • Operators behind residential proxies, who attach proxies per account to spread requests and reduce bans.
  • One-off CLI users exporting cookies with unjar x.com -f header | twscrape add_cookie my_account, then running twscrape search without writing any Python.
  • Python services that need raw payloads for custom parsing, calling api.search_raw(...) and the other _raw methods.

Getting started

Install with pip install twscrape, then add accounts from cookies and run the CLI (twscrape accounts, twscrape search) or call the async API from Python. Browser-like TLS fingerprinting is optional: pip install "twscrape[curl]" with TWS_HTTP_BACKEND=curl.

How it compares

The one similar tool named is SNScrape: twscrape returns SNScrape-compatible parsed models while adding an account pool, automatic rotation, SQLite session storage, per-account proxies, and raw response access. That makes it a drop-in shape for code already expecting SNScrape-style models, with the account and rate-limit layer sitting underneath.

When to use it — and when not to

Running it yourself means owning an authorized account pool, a SQLite account database, and, where needed, per-account proxies; the login flow can also require retrieving email verification codes. Anyone who cannot accept that X/Twitter's Terms of Service discourage using multiple accounts, or who objects to the README's referral links for third-party cookie-account and proxy providers, should look elsewhere. The project is community-maintained with a donation link rather than commercial support, and it carries 79 open issues.

project readme (upstream, from github) — read inline

twscrape

version py versions downloads license donate

twscrape is an async Python library and CLI for X/Twitter Search and GraphQL endpoints. It runs on your own account pool, keeps sessions in SQLite, rotates accounts when an endpoint is rate-limited, and returns either parsed SNScrape-style models or raw API responses.

Install

pip install twscrape

httpx is the default HTTP backend. For browser-like TLS fingerprinting, install the optional curl-cffi backend:

pip install "twscrape[curl]"

TWS_HTTP_BACKEND=curl twscrape user_by_login xdevelopers

Features

  • Search and GraphQL X/Twitter API methods
  • Async/await API for running multiple scrapers concurrently
  • Login flow with optional email verification code retrieval
  • Cookie-based account setup
  • Saved account sessions and per-account proxies
  • Raw Twitter API responses and parsed SNScrape-compatible models
  • Automatic account switching across rate-limited operations

Sponsor

RapidProxy is a residential proxy platform with 90M+ real IPs across 200+ countries. It supports rotation, geo-targeting, and high concurrency to improve scraping success and reduce bans. Start your free trial today!

Discount Code: RAPID10 to get 10% off.

Start With Cookies

twscrape requires authorized X/Twitter accounts. The most stable setup is to add an account from browser cookies containing auth_token and ct0. The recommended way to export them from your current browser profile is unjar:

unjar x.com -f header | twscrape add_cookie my_account
twscrape accounts
twscrape search "from:xdevelopers lang:en" --limit=20

my_account is a local identifier; twscrape does not verify that it matches the X username stored in the cookies. Run the same command again to replace its saved session while preserving credentials, statistics, locks, and proxy settings.

Alternatively, let the CLI prompt securely for cookies copied from x.com -> DevTools (F12) -> Application -> Cookies:

twscrape add_cookie my_account

Cookie accounts that include auth_token and ct0 are activated immediately; no login_accounts step is needed.

Ready-to-use cookie accounts are available from this provider. Proxy users can bring their own proxies or use this provider. These are referral links.

X/Twitter's Terms of Service discourage using multiple accounts. Use this project responsibly and at your own discretion.

Python API

import asyncio
from twscrape import API, gather


async def main():
    api = API()  # or API("accounts.db")

    # Add once; the session is stored in the account database.
    await api.pool.add_account_cookies("my_account", "auth_token=xxx; ct0=yyy")

    user = await api.user_by_login("xdevelopers")
    print(user.id, user.username, user.followersCount)

    tweets = await gather(api.search("from:xdevelopers lang:en", limit=20))
    for tweet in tweets:
        print(tweet.id, tweet.user.username, tweet.rawContent)


if __name__ == "__main__":
    asyncio.run(main())

gather() is a convenience helper. You can stream results directly:

async for tweet in api.search("open source lang:en", limit=100):
    print(tweet.id, tweet.rawContent)

Configure what happens when no account is immediately available:

api = API(raise_when_no_account=True, wait_timeout=30, wait_interval=1)

wait_timeout limits how long to wait for a locked account, wait_interval controls how often the pool checks again, and raise_when_no_account raises NoAccountError instead of ending the operation. By default, twscrape waits indefinitely while active accounts are locked.

Search defaults to the Latest tab. Pass kv={"product": "Top"} or kv={"product": "Media"} to use another search product:

tweets = await gather(api.search("python", limit=20, kv={"product": "Top"}))

Every parsed method has a _raw version for the original response wrapper:

async for rep in api.search_raw("from:xdevelopers", limit=20):
    print(rep.status_code, rep.json())

When breaking out of an async generator early, close it with contextlib.aclosing so the account lock is released promptly:

from contextlib import aclosing

async with aclosing(api.search("elon musk")) as gen:
    async for tweet in gen:
        if tweet.id < 200:
            break

API Surface

Search:

await gather(api.search("elon musk", limit=20))  # list[Tweet]
await gather(api.search("elon musk", limit=20, kv={"product": "Top"}))  # Top tab
await gather(api.search_user("openai", limit=20))  # list[User]
await gather(api.search_trend("python", limit=20))  # list[Trend]

Tweets:

tweet_id = 20

await api.tweet_details(tweet_id)  # Tweet
await gather(api.tweet_replies(tweet_id, limit=20))  # list[Tweet]
await gather(api.tweet_thread(tweet_id, limit=20))  # list[Tweet]
await gather(api.retweeters(tweet_id, limit=20))  # list[User]
await gather(api.bookmarks(limit=20))  # list[Tweet]

Users and timelines:

user_login = "xdevelopers"
user_id = 2244994945

await api.user_by_id(user_id)  # User
await api.user_by_login(user_login)  # User
await api.user_about(user_login)  # AccountAbout
await gather(api.following(user_id, limit=20))  # list[User]
await gather(api.followers(user_id, limit=20))  # list[User]
await gather(api.verified_followers(user_id, limit=20))  # list[User]
await gather(api.subscriptions(user_id, limit=20))  # list[User]
await gather(api.user_tweets(user_id, limit=20))  # list[Tweet]
await gather(api.user_tweets_and_replies(user_id, limit=20))  # list[Tweet]
await gather(api.user_media(user_id, limit=20))  # list[Tweet]

Lists:

list_id = 123456789

await gather(api.list_timeline(list_id, limit=20))  # list[Tweet]
await gather(api.list_members(list_id, limit=20))  # list[User]

Communities:

community_id = 1501272736215322629

await api.community_info(community_id)  # Community
await gather(api.community_members(community_id, limit=20))  # list[User]
await gather(api.community_moderators(community_id, limit=20))  # list[User]
await gather(api.community_tweets(community_id, limit=20))  # list[Tweet]

Trends:

await gather(api.trends("news"))  # list[Trend]
await gather(api.trends("sport"))  # list[Trend]
await gather(api.trends("entertainment"))  # list[Trend]
await gather(api.trends("VGltZWxpbmU6DAC2CwABAAAACHRyZW5kaW5nAAA"))  # list[Trend]

Parsed Tweet, User, Community, and trend objects can be converted with .dict() or .json().

CLI

twscrape
twscrape search --help

Commands:

twscrape search "QUERY" --limit=20
twscrape tweet_details TWEET_ID
twscrape tweet_replies TWEET_ID --limit=20
twscrape tweet_thread TWEET_ID --limit=20
twscrape retweeters TWEET_ID --limit=20
twscrape user_by_id USER_ID
twscrape user_by_login USERNAME
twscrape user_about USERNAME
twscrape user_media USER_ID --limit=20
twscrape following USER_ID --limit=20
twscrape followers USER_ID --limit=20
twscrape verified_followers USER_ID --limit=20
twscrape subscriptions USER_ID --limit=20
twscrape user_tweets USER_ID --limit=20
twscrape user_tweets_and_replies USER_ID --limit=20
twscrape list_timeline LIST_ID --limit=20
twscrape list_members LIST_ID --limit=20
twscrape community_info COMMUNITY_ID
twscrape community_members COMMUNITY_ID --limit=20
twscrape community_moderators COMMUNITY_ID --limit=20
twscrape community_tweets COMMUNITY_ID --limit=20
twscrape trends sport

CLI output is JSON Lines: one document per line.

twscrape search "elon musk lang:es" --limit=20 > tweets.jsonl
twscrape search "elon musk lang:es" --limit=20 --raw

Use a separate account database when you need isolated account pools:

twscrape --db research.db search "python lang:en" --limit=100

Accounts

Add username/password accounts from a file:

twscrape add_accounts ./accounts.txt username:password:email:email_password
twscrape login_accounts

twscrape login_accounts starts the login flow for each inactive account. If X asks for email verification and email_password is available, twscrape tries to read the code through IMAP and saves the resulting cookies for later use.

line_format describes how each line is split. Supported tokens:

  • username - required
  • password - required
  • email - required
  • email_password - used to fetch email verification codes through IMAP
  • cookies - cookie string, JSON, base64, or another format accepted by the parser
  • _ - skip column

Example account file:

username:password:email:email password:user_agent:cookies

Command:

twscrape add_accounts ./accounts.txt username:password:email:email_password:_:cookies

If IMAP is unavailable, enter verification codes manually:

twscrape login_accounts --manual
twscrape relogin user1 user2 --manual
twscrape relogin_failed --manual

Inspect and maintain the pool:

twscrape accounts
twscrape stats
twscrape relogin user1 user2
twscrape relogin_failed
twscrape reset_locks
twscrape delete_inactive
twscrape del_accounts user1 user2

twscrape accounts prints the current account state:

username  logged_in  active  last_used            total_req  error_msg
user1     True       True    2023-05-20 03:20:40  100        None
user2     True       True    2023-05-20 03:25:45  120        None
user3     False      False   None                 120        Login error

Limits

limit is the target number of parsed objects, not a page size. X/Twitter controls page size per endpoint, so a call can return fewer or more objects than requested.

Rate limits are tracked per account and per endpoint. When an account is limited for one operation, twscrape locks it for that operation until the reset time and tries another active account.

user_tweets and user_tweets_and_replies are limited by X/Twitter to about 3200 tweets.

Proxy

Use a proxy per account, per API instance, or for CLI commands:

await api.pool.add_account(
    "user1",
    "pass1",
    "user1@example.com",
    "email_pass1",
    proxy="http://login:pass@example.com:8080",
)

api = API(proxy="http://login:pass@example.com:8080")
doc = await api.user_by_login("xdevelopers")

TWS_PROXY=socks5://user:pass@127.0.0.1:1080 twscrape user_by_login xdevelopers

Proxy priority:

  1. api.proxy
  2. TWS_PROXY
  3. account proxy

Do not set api.proxy or TWS_PROXY when you want per-account proxies to be used.

Environment

  • TWS_PROXY - global proxy for all accounts
  • TWS_WAIT_EMAIL_CODE - email verification timeout in seconds, default 30
  • TWS_RAISE_WHEN_NO_ACCOUNT - raise NoAccountError instead of waiting; accepts false, 0, true, 1
  • TWS_HTTP_BACKEND - httpx or curl
  • TWS_LOG_LEVEL - logger level, default INFO
  • TWS_TELEMETRY=0 - disable anonymous telemetry

Telemetry

twscrape collects anonymous, aggregated telemetry about used GraphQL operation names and the selected HTTP backend. It does not collect usernames, cookies, proxies, queries, request URLs, or response bodies. Disable it with TWS_TELEMETRY=0.

See Also

Frequently asked questions

Is twscrape free to use?

twscrape is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does twscrape do?

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

What is twscrape written in?

twscrape is primarily written in Python. Its source is publicly available at https://github.com/vladkens/twscrape, and it has 2,915 GitHub stars.