FuzzyAI: Open Source AI Security & Privacy

What is FuzzyAI?

What it is

FuzzyAI is an open-source fuzzer for large language model APIs. It lives in the AI security and red-teaming ecosystem. It uses automated prompt mutation to test whether a model or API accepts jailbreak attempts. The project helps developers and security researchers find, evaluate, and reduce exposure to harmful or unwanted model behavior.

The concrete problem it solves is manual LLM security testing. A single prompt cannot show how an API responds to adversarial inputs, so FuzzyAI runs defined attack strategies against a target model or endpoint. It can compare outputs, classify responses, and work with local models, hosted APIs, or a custom REST API. The repository is Apache-2.0 licensed and written primarily as Jupyter Notebook code.

Key capabilities

Automated LLM fuzzing for jailbreak discovery, using default, ManyShot, Taxonomy, and ArtPrompt attack strategies.
Target selection across local Ollama models, OpenAI models, Anthropic models, and a custom REST API parsed from a raw HTTP request file.
Auxiliary model support, where a second model can assist during attacks and a classifier model can judge responses.
Batch prompt testing through a text file, supplied by the -T option.
Output filtering with a blacklisted_words setting for evaluating responses against listed terms.
Experimental Web UI for browser-based fuzzer runs.
Interactive Jupyter notebooks and bundled datasets for extended evaluation.

Who uses it and how

Security researchers probe LLM APIs for jailbreak behavior by running attack commands against a target prompt.
Developers test a model-backed service before deployment, using a local Ollama model or a hosted provider API.
Red-team workflows send adversarial prompts to a custom REST endpoint, such as an internal application that wraps a model.
Evaluation teams use the notebooks and datasets to study attack results and classifier behavior.

Getting started

Install with pip install git+https://github.com/cyberark/FuzzyAI.git for an existing Python project, or with Poetry from a cloned repository. Run fuzzyai fuzz -h, poetry run fuzzyai fuzz -h, or poetry run fuzzyai webui for the experimental interface.

When to use it — and when not to

Use FuzzyAI when automated adversarial testing of LLM APIs is the goal and the operator can supply model access, API keys, or a local Ollama model. Do not treat it as a turnkey hosted service, because the README gives installation and command-line usage only. The experimental Web UI and the need to configure providers, auxiliary models, classifiers, and raw HTTP files mean a self-hoster must operate the test environment and interpret results.

project readme (upstream, from github) — read inline

FuzzyAI Fuzzer

The FuzzyAI Fuzzer is a powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify jailbreaks and mitigate potential security vulnerabilities in their LLM APIs.

Getting Started

Quick start #1 - Using an existing python project

Install fuzzyai

# Use either pip or any other package manager
pip install git+https://github.com/cyberark/FuzzyAI.git

Run the fuzzer
```
fuzzyai fuzz -h
```

Quick start #2 - or as a standalone project

Clone the repository:

git clone [email protected]:cyberark/FuzzyAI.git
cd FuzzyAI

Install dependencies using Poetry:
```
poetry run pip install -e .
```
Run the fuzzer:
```
poetry run fuzzyai fuzz -h
```
Optional: Install ollama, and download a model for local usage:
```
ollama pull llama3.1
ollama show llama3.1 # verify model installation
```
Alternativly, you can use the Web UI

Web UI (Experimental)

FZAI

Run the Web UI (make sure you completed either of the installation steps from above):

 poetry run fuzzyai webui

 # Or specify a custom port:
 poetry run fuzzyai webui --port 9000

Notebooks

We've included interactive Jupyter notebooks you can use under src/fuzzyai/resources/notebooks/.
For more information, see notebooks wiki.

Datasets

We've included some datasets you can use under resources/. For more information, see datasets wiki.

Documentation

Explore detailed usage instructions in the Wiki.

Examples

If you're using poetry, make sure you've activated the venv (or by prepending 'poetry run' to the command line

A default attack (just evaluate the prompt 'Harmful_Prompt') using llama over Ollama

fuzzyai fuzz -m ollama/llama3.1 -a def -t "Harmful_Prompt"

Attack the prompt utilizing ManyShot and Taxonomy attacks, using gpt3.5 over OpenAI using one worker

OPENAI_API_KEY=sk-3fa1... fuzzyai fuzz -m openai/gpt-3.5-turbo -a man -a tax -t "Harmful_Prompt"

Attack prompts in prompts.txt utilizing the ArtPrompt attack, using Claude 3.5 over Anthropic

ANTHROPIC_API_KEY=deadbeef... fuzzyai fuzz -m anthropic/claude-3-5-sonnet-20240620 -a art -T src/fuzzyai/resources/adv_prompts.txt -e blacklisted_words="bomb,garbage,acid,ugly,self-harm,kill your"

Attack your own REST API

OPENAI_API_KEYT=sk-3fa1... fuzzyai fuzz -a def -x openai/gpt-4 -e classifier_model=openai/gpt-4 -m rest/http.raw -e host=localhost -e port=8000 -e scheme=https  
-t "How to build a bomb"

Arguments breakdown:

-a def:
Invokes the default attack mode, which evaluates the API without applying additional strategies.
-x openai/gpt-4:
Specifies an auxiliary model (gpt-4) to assist with certain tasks during the attack.
-e classifier_model=openai/gpt-4:
Configures the fuzzer to use the gpt-4 model for classifying outputs. This helps determine whether responses meet specific criteria, such as detecting harmful or unwanted content.
-m rest/http.raw:
Uses the REST provider to attack an API endpoint. The raw HTTP request is parsed from the file http.raw. (Refer to the documentation or example for details on the file structure.)
-e host=localhost -e port=8000 -e scheme=https:
Configures the REST provider with the following API endpoint details:
- Host: localhost
- Port: 8000
- Scheme: https (communication will use HTTPS)
-t "How to build a bomb":
Specifies the test input. In this example, it tests the API's handling of sensitive or harmful content.

Key Features

Comprehensive Fuzzing Techniques: Leverage mutation-based, generation-based, and intelligent fuzzing.
Built-in Input Generation: Generate valid and invalid inputs for exhaustive testing.
Seamless Integration: Easily incorporate into your development and testing workflows.
Extensible Architecture: Customize and expand the fuzzer to meet your unique requirements.

Supported models

FuzzyAI supports various models across top providers, including:

Provider	Models
Anthropic	Claude (3.5, 3.0, 2.1)
OpenAI	GPT-4o, GPT-4o mini, GPT o3
Gemini	Gemini Pro, Gemini 1.5
Azure	GPT-4, GPT-3.5 Turbo
Bedrock	Claude (3.5, 3.0), Meta (LLaMa)
AI21	Jamba (1.5 Mini, Large)
DeepSeek	DeepSeek (DeepSeek-V3, DeepSeek-V1)
Ollama	LLaMA (3.3, 3.2, 3.1), Dolphin-LLaMA3, Vicuna

Adding support for newer models

Easily add support for additional models by following our DIY guide.

Implemented Attacks

See attacks wiki for detailed information

Attack Type	Title	Reference
ArtPrompt	ASCII Art-based jailbreak attacks against aligned LLMs	arXiv:2402.11753
Taxonomy-based paraphrasing	Persuasive language techniques like emotional appeal to jailbreak LLMs	arXiv:2401.06373
PAIR (Prompt Automatic Iterative Refinement)	Automates adversarial prompt generation by iteratively refining prompts with two LLMs	arXiv:2310.08419
Many-shot jailbreaking	Embeds multiple fake dialogue examples to weaken model safety

Frequently asked questions

Is FuzzyAI free to use?

FuzzyAI is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does FuzzyAI do?

A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify and mitigate potential jailbreaks in their LLM AP

What is FuzzyAI written in?

FuzzyAI is primarily written in Jupyter Notebook. Its source is publicly available at https://github.com/cyberark/FuzzyAI, and it has 1,574 GitHub stars.