Promptfoo is an open-source command-line tool and TypeScript library for evaluating and red-teaming large language model applications, aimed at developers, AI engineers, and security teams who need measurable evidence that a prompt, agent, or retrieval-augmented generation pipeline behaves correctly and safely before it reaches users.

What it is

Promptfoo lives in the AI and machine learning tooling ecosystem as an evaluation and red-teaming framework for LLM applications. It is distributed as a CLI and as a library, written in TypeScript and licensed under MIT, with the README describing it as a tool for testing prompts, agents, and RAGs. It is used for automated evaluations, vulnerability scanning, and side-by-side model comparison across providers including OpenAI, Anthropic, Azure, Bedrock, and Ollama.

The concrete problem it replaces is trial-and-error prompt development. Instead of changing a prompt and guessing whether the output improved, a user writes a declarative configuration and runs promptfoo eval to produce scored results, then promptfoo view to inspect them. The same framework covers security: red teaming runs generate security vulnerability reports rather than subjective impressions. Evaluations run locally, so prompts do not leave the machine. Promptfoo is now part of OpenAI and remains open source under the MIT licence, according to the README.

Key capabilities

• Automated evaluations of prompts and models, driven by declarative configuration files and executed with promptfoo eval.
• Red teaming and vulnerability scanning for LLM applications, producing security vulnerability reports through the red-team workflow.
• Side-by-side model comparison across providers such as OpenAI, Anthropic, Azure, Bedrock, and Ollama.
• CI/CD integration for automated checks, with dedicated documentation for pipeline usage.
• Code scanning that reviews pull requests for LLM-related security and compliance issues.
• Command-line and library usage, including a Node.js package for programmatic invocation.
• Developer-oriented runtime features such as live reload and caching.

Who uses it and how

• Application teams running LLM features in production use it to gate prompt changes in CI/CD rather than shipping on intuition; the README states it powers LLM apps serving 10M+ users.
• Security and compliance engineers run red-team scans to surface vulnerabilities before release.
• Platform engineers compare candidate models from different providers on the same test suite to choose one on metrics instead of gut feel.
• Maintainers of RAG and agent pipelines use code scanning on pull requests to catch LLM-related security and compliance issues.
• Teams that require privacy run evaluations locally with their own API keys, keeping prompts on their own machines.

Getting started

Install with npm install -g promptfoo, brew install promptfoo, or pip install promptfoo, or run without installing via npx promptfoo@latest. Then create a project with promptfoo init --example getting-started, export the provider key such as OPENAI_API_KEY, and run promptfoo eval followed by promptfoo view inside the example directory.

How it compares

No list of paid products replaced by this project is provided in the facts, and no comparable tools are named. Promptfoo therefore stands alone in this registry.

When to use it — and when not to

Adopt it when a team needs repeatable, locally executed evals and red-team scanning wired into CI/CD. A self-hoster must operate a Node.js runtime of at least 22.22.0 for npm and npx usage, with Node.js 24 LTS recommended, and must supply API keys for most LLM providers. Teams that cannot manage Node.js runtimes or provider credentials, and anyone expecting a hosted service with no local setup, should look elsewhere; the project also carries a sizeable backlog of 651 open issues.

npm install -g promptfoo
promptfoo init --example getting-started

export OPENAI_API_KEY=sk-abc123

cd getting-started
promptfoo eval
promptfoo view