[opensourcedex]

garak is a free, open source ai security & privacy project written in Python and released under Apache-2.0. It has 9,285 GitHub stars, 1,290 forks and 429 open issues, and was last pushed 25 hours ago. On this registry it ranks #6 of 34 tracked projects in AI Security & Privacy, with 5 head-to-head comparisons available. It gained 27 stars over the last 3 tracked days.

What is garak?

garak is a free, Apache-2.0 command-line vulnerability scanner for large language models that probes generative AI systems for hallucination, data leakage, prompt injection, misinformation, toxicity generation, and jailbreaks, and it is aimed at security engineers, red teamers, and AI developers who need to test the models they ship.

What it is

garak stands for Generative AI Red-teaming and Assessment Kit. It is a Python command-line tool, licensed under Apache-2.0, developed for Linux and OSX. It combines static, dynamic, and adaptive probes to explore the ways an LLM or dialog system can be made to fail. In its own framing, garak checks whether an LLM can be made to fail in a way that is not wanted, and it treats that question as a scanning problem rather than a one-off manual review. It lives in the AI and machine learning security space, alongside the broader AI security and privacy tooling category.

The concrete problem it solves is the absence of a repeatable, automated way to red-team a model. A team deploying a generative model normally has no structured inventory of what it will do under adversarial pressure: which prompts extract training data, which ones bypass safety instructions, which ones induce fabricated output. garak supplies that structure. It replaces ad hoc, hand-written red-team prompt lists and one-off scripts with a probe-and-detector pipeline that can be pointed at a model and run end to end.

Key capabilities

  • garak --list_probes lists the available probes; by default garak runs all probes it knows against the target model, using the vulnerability detectors recommended by each probe.
  • --target_type and --target_name select the generator, for example --target_type huggingface --target_name "RWKV/rwkv-4-169m-pile".
  • Supported backends include Hugging Face Hub generative models, Replicate text models, the OpenAI API for both chat and continuation models, AWS Bedrock foundation models, LiteLLM, pretty much anything reachable over REST, and GGUF models such as llama.cpp version 1046 or later.
  • Probe coverage spans hallucination, data leakage, prompt injection, misinformation, toxicity generation, and jailbreaks.
  • Probe strategy combines static, dynamic, and adaptive techniques rather than relying on a fixed prompt set alone.
  • Installation paths include PyPI, a git development build, and a dedicated Conda environment with python>=3.11.

Who uses it and how

  • Security engineers running pre-deployment red-team passes against a generative model before it reaches users.
  • AI developers evaluating a model already selected for a product, who need evidence of failure modes rather than benchmark scores.
  • Teams standardized on OpenAI API, AWS Bedrock, or self-hosted GGUF and llama.cpp models can drive all of them through the same CLI with only --target_type and --target_name changing.
  • Groups hosting models on Hugging Face Hub or Replicate can point garak straight at the hosted model name.
  • The project is documented in arXiv paper 2406.11036, with community support through Discord at discord.gg/uVch4puUCs and reference material in the DEF CON slides at garak.ai.

Getting started

Standard install is python -m pip install -U garak; a fresher build comes from python -m pip install -U git+https://github.com/NVIDIA/garak.git@main, and a Conda environment can be created with python>=3.11. Documentation lives at docs.garak.ai and the project home is garak.ai.

How it compares

The two tools garak names for itself are nmap and Metasploit Framework. The analogy is direct: those tools scan and exercise networks and systems for weaknesses, and garak does the same job for LLMs and dialog systems. It is a free, Apache-2.0 project rather than a commercial scanning product, so there is no licence fee and no vendor-hosted console implied by the facts given here.

When to use it — and when not to

Choose garak when the goal is an automated, repeatable scan of a model across a known set of failure categories, and when the model is reachable through one of the supported backends or over REST. Do not choose it if the requirement is a managed dashboard, a hosted service, or a non-technical user interface; the facts describe a command-line tool only, with no hosted offering. The honest caveats are that the project carries 429 open issues at the time of writing, and that the README material describing the Conda setup and the generator catalogue is partly truncated, so some setup details need to be confirmed against docs.garak.ai rather than the README alone.

project readme (upstream, from github) — read inline

garak, LLM vulnerability scanner

Generative AI Red-teaming & Assessment Kit

garak checks if an LLM can be made to fail in a way we don't want. garak probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses. If you know nmap or msf / Metasploit Framework, garak does somewhat similar things to them, but for LLMs.

garak focuses on ways of making an LLM or dialog system fail. It combines static, dynamic, and adaptive probes to explore this.

garak's a free tool. We love developing it and are always interested in adding functionality to support applications.

License Tests/Linux Tests/Windows Tests/OSX Documentation Status arXiv discord-img Code style: black PyPI - Python Version PyPI Downloads Downloads

Get started

> See our user guide! docs.garak.ai

> Join our Discord!

> Project links & home: garak.ai

> Twitter: @garak_llm

> DEF CON slides!

LLM support

currently supports:

Install:

garak is a command-line tool. It's developed in Linux and OSX.

Standard install with pip

Just grab it from PyPI and you should be good to go:

python -m pip install -U garak

Install development version with pip

The standard pip version of garak is updated periodically. To get a fresher version from GitHub, try:

python -m pip install -U git+https://github.com/NVIDIA/garak.git@main

Clone from source

garak has its own dependencies. You can to install garak in its own Conda environment:

conda create --name garak "python>=3.11,<=3.13"
conda activate garak
gh repo clone NVIDIA/garak
cd garak
python -m pip install -e .

OK, if that went fine, you're probably good to go!

Note: if you cloned before the move to the NVIDIA GitHub organisation, but you're reading this at the github.com/NVIDIA URI, please update your remotes as follows:

git remote set-url origin https://github.com/NVIDIA/garak.git

Getting started

The general syntax is:

garak

garak needs to know what model to scan, and by default, it'll try all the probes it knows on that model, using the vulnerability detectors recommended by each probe. You can see a list of probes using:

garak --list_probes

To specify a generator, use the --target_type and, optionally, the --target_name options. Model type specifies a model family/interface; model name specifies the exact model to be used. The "Intro to generators" section below describes some of the generators supported. A straightforward generator family is Hugging Face models; to load one of these, set --target_type to huggingface and --target_name to the model's name on Hub (e.g. "RWKV/rwkv-4-169m-pile"). Some generators might need an API key to be set as an environment variable, and they'll let you know if they need that.

garak runs all the probes by default, but you can be specific about that too. --spec probes.promptinject will use only the PromptInject framework's methods, for example. You can also specify one specific plugin instead of a plugin family by adding the plugin name after a .; for example, --spec probes.lmrc.SlurUsage will use an implementation of checking for models generating slurs based on the Language Model Risk Cards framework.

For help and inspiration, find us on Twitter or discord!

Examples

Probe a commercial model for encoding-based prompt injection (OSX/*nix) (replace example value with a real OpenAI API key)

export OPENAI_API_KEY="sk-123XXXXXXXXXXXX"
python3 -m garak --target_type openai --target_name gpt-5-nano --spec probes.encoding

See if the Hugging Face version of GPT2 is vulnerable to DAN 11.0

python3 -m garak --target_type huggingface --target_name gpt2 --spec probes.dan.Dan_11_0

Reading the results

For each probe loaded, garak will print a progress bar as it generates. Once generation is complete, a row evaluating that probe's results on each detector is given. If any of the prompt attempts yielded an undesirable behavior, the response will be marked as FAIL, and the failure rate given.

Here are the results with the encoding module on a GPT-3 variant: alt text

And the same results for ChatGPT: alt text

We can see that the more recent model is much more susceptible to encoding-based injection attacks, where text-babbage-001 was only found to be vulnerable to quoted-printable and MIME encoding injections. The figures at the end of each row, e.g. 840/840, indicate the number of text generations total and then how many of these seemed to behave OK. The figure can be quite high because more than one generation is made per prompt - by default, 10.

Errors go in garak.log; the run is logged in detail in a .jsonl file specified at analysis start & end. There's a basic analysis script in analyse/analyse_log.py which will output the probes and prompts that led to the most hits.

Send PRs & open issues. Happy hunting!

Intro to generators

Hugging Face

Using the Pipeline API:

  • --target_type huggingface (for transformers models to run locally)
  • --target_name - use the model name from Hub. Only generative models will work. If it fails and shouldn't, please open an issue and paste in the command you tried + the exception!

Using the Inference API:

  • --target_type huggingface.InferenceAPI (for API-based model access)
  • --target_name - the model name from Hub, e.g. "mosaicml/mpt-7b-instruct"

Using private endpoints:

  • --target_type huggingface.InferenceEndpoint (for private endpoints)

  • --target_name - the endpoint URL, e.g. https://xxx.us-east-1.aws.endpoints.huggingface.cloud

  • (optional) set the HF_INFERENCE_TOKEN environment variable to a Hugging Face API token with the "read" role; see https://huggingface.co/settings/tokens when logged in

OpenAI

  • --target_type openai
  • --target_name - the OpenAI model you'd like to use. gpt-5-nano is fast and fine for testing.
  • set the OPENAI_API_KEY environment variable to your OpenAI API key (e.g. "sk-19763ASDF87q6657"); see https://platform.openai.com/account/api-keys when logged in

Recognised model types are whitelisted, because the plugin needs to know which sub-API to use. Completion or ChatCompletion models are OK. If you'd like to use a model not supported, you should get an informative error message, and please send a PR / open an issue.

Replicate

Public Replicate models:

  • --target_type replicate
  • --target_name - the Replicate model name and hash, e.g. "stability-ai/stablelm-tuned-alpha-7b:c49dae36"

Private Replicate endpoints:

  • --target_type replicate.InferenceEndpoint (for privat

readme truncated — read the full docs on github

Frequently asked questions

Is garak free to use?

garak is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does garak do?

the LLM vulnerability scanner

What is garak written in?

garak is primarily written in Python. Its source is publicly available at https://github.com/NVIDIA/garak, and it has 9,285 GitHub stars.