cariddi: Open Source Data Extraction & Web Scraping

What is cariddi?

Cariddi is an open-source, GPL-3.0 command-line crawler and scanner written in Go that takes a list of domains on standard input, crawls the pages it finds, and reports endpoints, secrets, API keys, tokens, file extensions, errors and other useful information, built for bug bounty hunters, penetration testers and OSINT practitioners.

What it is

Cariddi is a single Go binary maintained by edoardottt at github.com/edoardottt/cariddi. It reads targets from standard input, either one URL at a time or a file such as urls.txt containing many lines, then crawls those targets and applies whichever scan modes the operator selects. The project sits in the infosec and offensive-security toolchain, tagged with bugbounty, pentesting, penetration-testing, infosec, osint, endpoint-discovery and crawler. It carries 3,772 stars, 344 forks and 14 open issues, with the most recent push in September 2026.

The concrete problem it solves is the manual loop that reconnaissance work normally requires: walking a target with a browser or routing traffic through an intercepting proxy, then reading through captured responses by eye to spot exposed endpoints, credentials and tokens. Cariddi automates the crawl and the matching in one pass, so a scope list goes in and a structured list of findings comes out. Results can be printed plainly for pipelines, written to a text file, rendered as HTML, or emitted as JSON on stdout.

Key capabilities

Dedicated scan modes: -s hunts secrets, -e hunts juicy endpoints, -err hunts errors, -info hunts useful information, and -intensive crawls while also searching subdomains in the *.target.com shape.
Graded file-extension hunting with -ext 2, which targets level 2 out of 7, plus an ignore list for assets such as png, svg, jpg, jpeg, woff, ttf, mp4, mp3 and ico that is skipped while scanning for secrets, info and errors unless overridden with -ie pdf,png,jpg.
Custom detection patterns through -ef endpoints_file and -sf secrets_file, letting operators extend the built-in endpoint and secret sets.
Request control with -proxy http://127.0.0.1:8080 for both HTTP and SOCKS5, -c 200 for concurrency, -d 2 for a delay between crawled pages, -t 5 for request timeout, and -md 3 for maximum crawl depth.
Identity and header handling with -ua "Custom User Agent", -rua for a random browser user agent per request, -headers for inline custom headers, and -headersfile headers.txt for headers read from a file.
URL filtering with -i forum,blog,community,open to drop URLs containing given words, and -it ignore_file to drop URLs matching any line in a file.
Output and caching options: -plain for results only, -ot target_name for a text file, -oh target_name for an HTML file, -json for machine-readable stdout, -sr to store HTTP responses, -debug for crawl diagnostics, and -cache to use the .cariddi_cache folder.

Who uses it and how

Bug bounty hunters feed a scope list from reconnaissance output straight into cariddi to surface leaked keys, tokens and API endpoints before manual triage begins.
Penetration testers route crawls through an intercepting proxy with -proxy so traffic lands in existing tooling, and keep raw responses with -sr for evidence.
OSINT and infosec analysts tune -i, -it and -d to keep a crawl narrow and paced against targets they do not own.
Automation-friendly workflows rely on the stdin and stdout design, using -json or -plain for pipelines and -ot or -oh for report artifacts.
Windows operators run it as cat urls.txt | cariddi.exe from PowerShell, with the executable kept inside the cariddi folder.

Getting started

Install through Homebrew with brew install cariddi, Snap with sudo snap install cariddi, or Go with go install -v github.com/edoardottt/cariddi/cmd/cariddi@latest; Pacman and NixOS packages also exist. Then run echo https://edoardottt.com/ | cariddi, or pipe a urls.txt file in.

How it compares

No paid products and no comparable tools are named in the provided facts, so cariddi stands alone in this registry entry.

When to use it — and when not to

A self-hoster operates nothing but the binary: targets come from the operator's own list, caching writes to the local .cariddi_cache folder, and there is no server component, database or mail service to run. The GPL-3.0 licence means anyone redistributing a modified build takes on copyleft obligations. Teams wanting a hosted dashboard, a graphical interface or scheduled cloud scanning should look elsewhere, and operators should be careful running combined scan modes at high -c values against production hosts, since the crawl is only as polite as the -d delay they set.

project readme (upstream, from github) — read inline

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

_{Coded with 💙 by edoardottt}

Share on Twitter!

Install • Usage • Get Started • Changelog • Contributing • License

Installation 📡

Homebrew

brew install cariddi

Snap

sudo snap install cariddi

Golang

go install -v github.com/edoardottt/cariddi/cmd/cariddi@latest

Pacman

pacman -Syu cariddi

NixOS

nix-shell -p cariddi

Building from source

You need Go (>=1.24.0)

Building from source for Linux and Windows

Linux

git clone https://github.com/edoardottt/cariddi.git
cd cariddi
go get ./...
make linux # (to install)
make unlinux # (to uninstall)

One-liner: git clone https://github.com/edoardottt/cariddi.git && cd cariddi && go get ./... && make linux

Windows

Note that the executable works only in cariddi folder.

git clone https://github.com/edoardottt/cariddi.git
cd cariddi
go get ./...
.\make.bat windows # (to install)
.\make.bat unwindows # (to uninstall)

Usage 💡

If you want to scan only a single target you can use

echo https://edoardottt.com/ | cariddi

With multiple targets you can use a file instead, e.g. urls.txt containing:

https://edoardottt.com/
http://testphp.vulnweb.com/

For Windows:

use powershell.exe -Command "cat urls.txt | .\cariddi.exe" inside the Command prompt
or just cat urls.txt | cariddi.exe using PowerShell

Basics

cariddi -version (Print the version)
cariddi -h (Print the help)
cariddi -examples (Print the examples)

Scan options

cat urls.txt | cariddi -intensive (Crawl searching also subdomains, same as *.target.com)
cat urls.txt | cariddi -s (Hunt for secrets)
cat urls.txt | cariddi -err (Hunt for errors in websites)
cat urls.txt | cariddi -e (Hunt for juicy endpoints)
cat urls.txt | cariddi -info (Hunt for useful informations in websites)
cat urls.txt | cariddi -ext 2 (Hunt for juicy (level 2 out of 7) files)
cat urls.txt | cariddi -e -ef endpoints_file (Hunt for custom endpoints)
cat urls.txt | cariddi -s -sf secrets_file (Hunt for custom secrets)
cat urls.txt | cariddi -ie pdf,png,jpg (Ignore these extensions while scanning)

Default: png, svg, jpg, jpeg, bmp, jfif, gif, webp, woff, woff2, ttf, tiff, tif, mp4, webm, mkv, avi, mov, flv, wmv, mp3, wav, flac, ogg, m4a, aac, ico, cur, eot, otf are ignored while scanning for secrets, info and errors.

Configuration

cat urls.txt | cariddi -proxy http://127.0.0.1:8080 (Set a Proxy, http and socks5 supported)
cat urls.txt | cariddi -d 2 (2 seconds between a page crawled and another)
cat urls.txt | cariddi -c 200 (Set the concurrency level to 200)
cat urls.txt | cariddi -i forum,blog,community,open (Ignore urls containing these words)
cat urls.txt | cariddi -it ignore_file (Ignore urls containing at least one line in the input file)
cat urls.txt | cariddi -cache (Use the .cariddi_cache folder as cache)
cat urls.txt | cariddi -t 5 (Set the timeout for the requests)
cat urls.txt | cariddi -headers "Cookie: auth=admin;type=2;; X-Custom: customHeader"
cat urls.txt | cariddi -headersfile headers.txt (Read from an external file custom headers)
cat urls.txt | cariddi -ua "Custom User Agent" (Use a custom User Agent)
cat urls.txt | cariddi -rua (Use a random browser user agent on every request)

Output

cat urls.txt | cariddi -plain (Print only results)
cat urls.txt | cariddi -ot target_name (Results in txt file)
cat urls.txt | cariddi -oh target_name (Results in html file)
cat urls.txt | cariddi -json (Print the output as JSON in stdout)
cat urls.txt | cariddi -sr (Store HTTP responses)
cat urls.txt | cariddi -debug (Print debug information while crawling)
cat urls.txt | cariddi -md 3 (Max 3 depth levels)

Get Started 🎉

cariddi -h prints the help.

Usage of cariddi:
  -c int
     Concurrency level. (default 20)
  -cache
     Use the .cariddi_cache folder as cache.
  -d int
     Delay between a page crawled and another.
  -debug
     Print debug information while crawling.
  -e Hunt for juicy endpoints.
  -ef string
     Use an external file (txt, one per line) to use custom parameters for endpoints hunting.
  -err
     Hunt for errors in websites.
  -examples
     Print the examples.
  -ext int
     Hunt for juicy file extensions. Integer from 1(juicy) to 7(not juicy).
  -h Print the help.
  -headers string
     Use custom headers for each request E.g. -headers "Cookie: auth=yes;;Client: type=2".
  -headersfile string
     Read from an external file custom headers (same format of headers flag).
  -json
     Print the output as JSON in stdout.
  -md
     Maximum depth level the crawler will follow from the initial target URL.
  -i string
     Ignore the URL containing at least one of the elements of this array.
  -ie value
     Comma-separated list of extensions to ignore while scanning.
  -info
     Hunt for useful informations in websites.
  -intensive
     Crawl searching for resources matching 2nd level domain.
  -it string
     Ignore the URL containing at least one of the lines of this file.
  -oh string
     Write the output into an HTML file.
  -ot string
     Write the output into a TXT file.
  -plain
     Print only the results.
  -proxy string
     Set a Proxy to be used (http and socks5 supported).
  -rua
     Use a random browser user agent on every request.
  -s Hunt for secrets.
  -sf string
     Use an external file (txt, one per line) to use custom regexes for secrets hunting.
  -sr
     Store HTTP responses.
  -t int
     Set timeout for the requests. (default 10)
  -ua string
     Use a custom User Agent.
  -version
     Print the version.

Click to understand How to integrate cariddi with Burpsuite

Normally you use Burpsuite within your browser, so you just have to trust the burpsuite's certificate in the browser and you're done.
In order to use cariddi with the BurpSuite proxy you should do some steps further.

If you try to use cariddi with the option -proxy http://127.0.0.1:8080 you will find this error in the burpsuite error log section:

Received fatal alert: bad_certificate (or something similar related to the certificate).

To make cariddi working fine with Burpsuite you have also to trust the certificate within your entire pc, not just only the browser. These are the steps you have to follow:

Go to Proxy tab in Bupsuite, then Options. Click on the CA Certificate button and export Certificate in DER format

openssl x509 -in burp.der -inform DER -out burp.pem -outform PEM
sudo chown root:root burp.pem
sudo chmod 644 burp.pem
sudo cp burp.pem /usr/local/share/ca-certificates/
sudo c_rehash
cd /etc/ssl/certs/
sudo ln -s /usr/local/share/ca-certificates/burp.pem
sudo c_rehash .

Source: Trust Burp Proxy certificate in Debian/Ubuntu

After these steps, in order to use cariddi with Burpsuite you have to:

Open Burpsuite, making sure that the proxy is listening.
Use cariddi with the flag -proxy http://127.0.0.1:8080.
You will see that requests and responses will be logged in Burpsuite.

Changelog 📌

Detailed changes for each release are documented in the release notes.

Contributing 🛠

Just open an issue/pull request.

Before opening a pull request, download golangci-lint and run

golangci-lint run

If there aren't errors, go ahead :)

Test using https://edoardottt.github.io/cariddi-test/

echo "https://edoardottt.github.io/cariddi-test/" | cariddi

Help me build this!

Special thanks to: go-colly, ocervell, zricethezav, projectdiscovery, tomnomnom, RegexPassive and all the contributors.

License 📝

This repository is under GNU General Public License v3.0.
edoardottt.com to contact me.

Frequently asked questions

Is cariddi free to use?

cariddi is open source under the GPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does cariddi do?

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

What is cariddi written in?

cariddi is primarily written in Go. Its source is publicly available at https://github.com/edoardottt/cariddi, and it has 3,811 GitHub stars.