colly: Open Source Data Extraction & Web Scraping

What is colly?

What it is

Colly is an open-source scraping and crawling framework written in Go, distributed under the Apache-2.0 license. It lives in the Go ecosystem and is consumed as a library rather than a standalone application: the README describes it as providing a clean interface for writing any kind of crawler, scraper, or spider, and the project's own topics list it as a crawler, crawling framework, scraper, scraping tool, and spider. The repository has been public for roughly nine years, carries 25,516 stars and 1,862 forks, and its most recent push was on 2 September 2026. Documentation and examples are hosted at go-colly.org, with the API reference published on pkg.go.dev.

The concrete problem it solves is the boilerplate that otherwise surrounds web data extraction in Go. Rather than hand-rolling HTTP clients, link traversal, concurrency limits, and response decoding, a developer registers callbacks such as OnHTML and OnRequest on a collector and lets the framework drive the crawl. The README frames the output as structured data suitable for data mining, data processing, or archiving, and the project's example set shows the same pattern applied to link discovery, request logging, and page visiting. It is a general-purpose extraction engine, not a service tied to one site or one data shape.

Key capabilities

Throughput above 1,000 requests per second on a single core, per the README's feature list.
Per-domain management of request delays and maximum concurrency.
Automatic cookie and session handling across requests.
Synchronous, asynchronous, and parallel scraping modes.
Response caching, plus automatic encoding of non-unicode responses.
Robots.txt support and configuration through environment variables.
A distributed scraping mode and an extension mechanism for adding behaviour.

Who uses it and how

Greenpeace's check-my-pages uses it as a scraping script to test the Spanish Greenpeace web archive.
JiveSearch, a search engine that does not track users, relies on it as part of its crawling stack.
gamedb builds a database of Steam games with it, and WeiboSpiderGo scrapes Sina Weibo.
goclone clones websites to a local machine, while gophie searches, downloads, and streams

project readme (upstream, from github) — read inline

Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

Example


import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}

See examples folder for more detailed examples.

Installation

go get github.com/gocolly/colly/v2

Bugs

Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

greenpeace/check-my-pages Scraping script to test the Spanish Greenpeace web archive.
altsab/gowap Wappalyzer implementation in Go.
jesuiscamille/goquotes A quotes scraper, making your day a little better!
jivesearch/jivesearch A search engine that doesn't track you.
Leagify/colly-draft-prospects A scraper for future NFL Draft prospects.
lucasepe/go-ps4 Search playstation store for your favorite PS4 games using the command line.
yringler/inside-chassidus-scraper Scrapes Rabbi Paltiel's web site for lesson metadata.
gamedb/gamedb A database of Steam games.
lawzava/scrape CLI for email scraping from any website.
eureka101v/WeiboSpiderGo A sina weibo(chinese twitter) scraper
Go-phie/gophie Search, Download and Stream movies from your terminal
imthaghost/goclone Clone websites to your computer within seconds.
superiss/spidy Crawl the web and collect expired domains.
docker-slim/docker-slim Optimize your Docker containers to make them smaller and better.
seversky/gachifinder an agent for asynchronous scraping, parsing and writing to some storages(elasticsearch for now)
eval-exec/goodreads crawl all tags and all pages of quotes from goodreads.

If you are using Colly in a project please send a pull request to add it to the list.

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! 🙏 [Become a backer]

License

Frequently asked questions

Is colly free to use?

colly is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does colly do?

Elegant Scraper and Crawler Framework for Golang

What is colly written in?

colly is primarily written in Go. Its source is publicly available at https://github.com/gocolly/colly, and it has 25,520 GitHub stars.