[opensourcedex]

katana is a free, open source data extraction & web scraping project written in Go and released under MIT. It has 17,518 GitHub stars, 1,183 forks and 23 open issues, and was last pushed 3 days ago. On this registry it ranks #14 of 45 tracked projects in Data Extraction & Web Scraping, with 5 head-to-head comparisons available. It gained 7 stars over the last 3 tracked days.

What is katana?

Katana is an open-source, MIT-licensed crawling and spidering framework written in Go, built for security engineers and automation pipelines that need to map web applications at scale rather than explore them by hand.

What it is

Katana is a command-line web crawler and spidering framework distributed under the MIT licence, written in Go, and developed in the ProjectDiscovery tooling ecosystem alongside its other security utilities. The README describes it as a fast crawler focused on execution in automation pipelines, and the project ships both a standard HTTP crawling mode and a headless browser mode so that the same tool can walk static pages and JavaScript-rendered applications. It carries the cli, crawler, gocrawler, headless, spider-framework and web-spider topics, and it has accumulated roughly 17.5k stars and 1.1k forks.

The concrete problem it solves is the crawling step of a reconnaissance or data-extraction workflow, the part that otherwise falls to a one-off script or a general-purpose browser automation harness per target. Katana folds input handling, scope control, JavaScript parsing, form discovery, filtering and structured output into one binary, so a scan can be driven from STDIN, a single URL passed with -u, or a target list passed with -list, and results emitted to STDOUT, a file, or JSON. It also supports resuming an interrupted run through -resume resume.cfg, which matters when a crawl runs for hours against a large target.

Key capabilities

  • Standard and headless crawling modes, with headless driven by -system-chrome -headless against a locally installed Chrome or Chromium binary.
  • JavaScript endpoint parsing and crawling via -jc, -js-crawl, plus optional jsluice parsing through -jsl, -jsluice, which the README flags as memory intensive.
  • Automatic form filling with -aff, -automatic-form-fill (experimental) and form extraction with -fx, -form-extraction, which emits form, input, textarea and select elements in jsonl output.
  • Scope control through -d, -depth (default 3), -ct, -crawl-duration in seconds, minutes, hours or days, -e, -exclude filters matching cdn, private-ips, CIDR, IP or regex, -fsu, -filter-similar and -iqp, -ignore-query-params.
  • Selectable traversal order with -s, -strategy, choosing between depth-first and breadth-first.
  • A knowledge base with machine-learning page-type and form classification, using an automatically downloaded model, plus technology detection through -td, -tech-detect.
  • Request control through -proxy for HTTP or SOCKS5, custom resolvers with -r, -resolvers, custom headers and cookies with -H, -headers, and -kf, -known-files to crawl robots.txt and sitemap.xml.

Who uses it and how

  • Security and bug-bounty teams running crawl stages inside larger recon pipelines, feeding targets in from STDIN and consuming JSON for downstream parsing.
  • CI and container environments using the projectdiscovery/katana Docker image, either in standard mode or with -system-chrome -headless for JavaScript-heavy targets.
  • Teams needing per-engagement behaviour, tuning -fc, -form-config and -flc, -field-config to match custom form and field definitions.
  • Long-running enumerations on large scopes, where -resume and duration-based termination keep partial progress usable.

Getting started

Install with CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest, which requires Go 1.26 or newer, or pull the prebuilt image with docker pull projectdiscovery/katana:latest and run docker run projectdiscovery/katana:latest -u https://tesla.com. Usage and the full flag list are available through katana -h.

How it compares

No paid or competing products are named in the facts provided, so this entry stands alone in this registry. Katana is MIT-licensed and self-hosted, with no vendor account, seat count or per-scan cost model attached to it. Comparisons with other crawlers therefore rest on deployment and licensing choices rather than on any product named here.

When to use it — and when not to

A self-hoster must supply Go 1.26 or newer with CGO_ENABLED=1 for source installs, and headless mode requires a locally installed Chrome package such as google-chrome-stable on Ubuntu. Anyone wanting a hosted crawling service, or a team unwilling to run a browser binary in its environment, should look elsewhere, and -jsl parsing should be avoided on memory-constrained hosts. Two caveats sit in the README itself: automatic form filling is labelled experimental, and crawling of known files requires a minimum depth of 3 to work correctly.

project readme (upstream, from github) — read inline

katana

A next-generation crawling and spidering framework

Features • Installation • Usage • Scope • Config • Filters • Join Discord

Features

image

  • Fast And fully configurable web crawling
  • Standard and Headless mode
  • JavaScript parsing / crawling
  • Customizable automatic form filling
  • Scope control - Preconfigured field / Regex
  • Knowledge base - ML page-type / form classification (auto-downloaded model)
  • Customizable output - Preconfigured fields
  • INPUT - STDIN, URL and LIST
  • OUTPUT - STDOUT, FILE and JSON

Installation

katana requires Go 1.26+ to install successfully. If you encounter any installation issues, we recommend trying with the latest available version of Go, as the minimum required version may have changed. Run the command below or download a pre-compiled binary from the release page.

CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest

More options to install / run katana-

Docker

To install / update docker to latest tag -

docker pull projectdiscovery/katana:latest

To run katana in standard mode using docker -

docker run projectdiscovery/katana:latest -u https://tesla.com

To run katana in headless mode using docker -

docker run projectdiscovery/katana:latest -u https://tesla.com -system-chrome -headless
Ubuntu

It's recommended to install the following prerequisites -

sudo apt update
sudo apt install zip curl wget git snapd
sudo snap refresh
sudo snap install golang --classic

sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://dl.google.com/linux/linux_signing_key.pub \
  | sudo gpg --dearmor -o /etc/apt/keyrings/google-chrome.gpg

echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/google-chrome.gpg] \
  http://dl.google.com/linux/chrome/deb/ stable main" \
  | sudo tee /etc/apt/sources.list.d/google-chrome.list > /dev/null

sudo apt update
sudo apt install google-chrome-stable

install katana -

go install github.com/projectdiscovery/katana/cmd/katana@latest

Usage

katana -h

This will display help for the tool. Here are all the switches it supports.

Katana is a fast crawler focused on execution in automation
pipelines offering both headless and non-headless crawling.

Usage:
  ./katana [flags]

Flags:
INPUT:
   -u, -list string[]     target url / list to crawl
   -resume string         resume scan using resume.cfg
   -e, -exclude string[]  exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)

CONFIGURATION:
   -r, -resolvers string[]       list of custom resolver (file or comma separated)
   -d, -depth int                maximum depth to crawl (default 3)
   -jc, -js-crawl                enable endpoint parsing / crawling in javascript file
   -jsl, -jsluice                enable jsluice parsing in javascript file (memory intensive)
   -ct, -crawl-duration value    maximum duration to crawl the target for (s, m, h, d) (default s)
   -kf, -known-files string      enable crawling of known files (all,robotstxt,sitemapxml), a minimum depth of 3 is required to ensure all known files are properly crawled.
   -mrs, -max-response-size int  maximum response size to read (default 4194304)
   -timeout int                  time to wait for request in seconds (default 10)
   -aff, -automatic-form-fill    enable automatic form filling (experimental)
   -fx, -form-extraction         extract form, input, textarea & select elements in jsonl output
   -retry int                    number of times to retry the request (default 1)
   -proxy string                 http/socks5 proxy to use
   -td, -tech-detect             enable technology detection
   -H, -headers string[]         custom header/cookie to include in all http request in header:value format (file)
   -config string                path to the katana configuration file
   -fc, -form-config string      path to custom form configuration file
   -flc, -field-config string    path to custom field configuration file
   -s, -strategy string          Visit strategy (depth-first, breadth-first) (default "depth-first")
   -iqp, -ignore-query-params    Ignore crawling same path with different query-param values
   -fsu, -filter-similar         filter crawling of similar looking URLs (e.g., /users/123 and /users/456)
   -fst, -filter-similar-threshold int  number of distinct values before a path position is treated as parameter (default 10)
   -tlsi, -tls-impersonate       enable experimental client hello (ja3) tls randomization
   -dr, -disable-redirects       disable following redirects (default false)
   -pcs, -page-content-similar   enable page content similarity filtering (simhash|tfidf|bm25)
   -pcsm, -page-content-similar-mode string  similarity mode: simhash, tfidf, or bm25 (default simhash)
   -pcsd, -page-content-similar-distance int  simhash max hamming distance (default 3)
   -pcst, -page-content-similar-threshold float  tfidf/bm25 min score 0-1 (default 0.85)
   -pcsn, -page-content-similar-budget int  pages to fully process per similarity cluster (default 1)
   -sdd, -similarity-deduplication  alias for -pcs
   -kb, -knowledge-base          enable knowledge base classification
   -kb-secrets                   enable secrets extractor in the knowledge base
   -kb-validate-secrets          validate detected secrets against their provider (sends live API calls)
   -kb-endpoints                 enable endpoints extractor (classifies REST/GraphQL/SOAP/XHR requests)
   -mdp, -max-domain-pages int   maximum number of pages to crawl per domain (default unlimited)

DEBUG:
   -health-check, -hc        run diagnostic check up
   -elog, -error-log string  file to write sent requests error log
   -pprof-server             enable pprof server

HEADLESS:
   -hl, -headless                    enable headless hybrid crawling (experimental)
   -sc, -system-chrome               use local installed chrome browser instead of katana installed
   -sb, -show-browser                show the browser on the screen with headless mode
   -ho, -headless-options string[]   start headless chrome with additional options
   -nos, -no-sandbox                 start headless chrome in --no-sandbox mode
   -cdd, -chrome-data-dir string     path to store chrome browser data
   -scp, -system-chrome-path string  use specified chrome browser for headless crawling
   -noi, -no-incognito               start headless chrome without incognito mode
   -cwu, -chrome-ws-url string       use chrome browser instance launched elsewhere with the debugger listening at this URL
   -xhr, -xhr-extraction             extract xhr request url,method in jsonl output
   -pls, -page-load-strategy string  page load strategy (heuristic, load, domcontentloaded, networkidle, none) (default "heuristic")
   -dwt, -dom-wait-time int          time in seconds to wait after page load when using domcontentloaded strategy (default 5)
   -csp, -captcha-solver-provider string  captcha solver provider (e.g. capsolver)
   -csk, -captcha-solver-key string       captcha solver provider api key

SCOPE:
   -cs, -crawl-scope string[]       in scope url regex to be followed by crawler
   -cos, -crawl-out-scope string[]  out of scope url regex to be excluded by crawler
   -fs, -field-scope string         pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn")
   -ns, -no-scope                   disables host based default scope
   -do, -display

readme truncated — read the full docs on github

Frequently asked questions

Is katana free to use?

katana is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does katana do?

A next-generation crawling and spidering framework.

What is katana written in?

katana is primarily written in Go. Its source is publicly available at https://github.com/projectdiscovery/katana, and it has 17,518 GitHub stars.