iFixAi

English · 简体中文 · 日本語 · 한국어

Independent Auditing of AI Agents

Catch your agent's mistakes and blind spots before the shit hits the fan.

Quick start • Three ways to run • Test your agent • Scoring • Docs • Contributing

One ifixai run, end to end: guided setup picks the system, judge, and suite; the run verifies the connection and saves your config; 32 inspections execute across five pillars; and the result lands as an A–F grade with a scored core-pillar scorecard.

What it is

The existing Eval, Red-teaming, and Observability Tools are evaluating the agent mainly based on tech capability (token efficiency, latency, prompt injections). They cannot answer the most crucial question.

Is the agent doing the job it is supposed to do based on the business KPIs and Organizational Structure? iFixAi gives you this answer in less than 120 seconds by striking the right balance between AI-Red Teaming and Operational Assurance.

Adversarial depth. Assurance discipline. All-in-one auditing process.

Three ways to run

All three run the same diagnostic underneath. The difference is how you configure and drive it.

Quick start

Now try it yourself. Pick a path from the table above; full walkthrough: docs/get-started.md.

Guided wizard (recommended)

pip install "ifixai[openai]"   # or anthropic, gemini, etc.: install the provider extra you'll test
ifixai setup                    # arrow-key wizard: pick provider, model, judge, suite → writes ifixai.yaml
ifixai run                      # no flags needed; reports land in ./ifixai-results/

ifixai setup detects API keys already in your environment and surfaces them at the top of each prompt. No key found? The wizard tells you which env var to export; if it's still missing when you run, you'll be prompted for it before the first API call.

Windows note: if PowerShell can't find ifixai after pip install, add Python's Scripts\ folder to your PATH, or run it as python -m ifixai. This is the usual Python-on-Windows PATH gap, not an iFixAi issue.

Plugin (Claude Code and Codex)

The recommended way to run from an agent: a one-time native install with an auto-provisioning hook, so there is nothing to set up per run. Ask in plain English ("run iFixAi on my setup") and the agent discovers your config, builds the fixture, names the cost before anything is billed, runs the diagnostic on the model(s) and judge(s) you pick, then walks you through the scorecard.

Claude Code, from inside Claude Code:

/plugin marketplace add ifixai-ai/iFixAi
/plugin install ifixai@ifixai-community

Then ask "run iFixAi on my setup", or type /ifixai:ifixai. (Restart Claude Code or run /reload-plugins if it doesn't appear.) Already on ifixai@ifixai-ai? That keeps working, and to move to the new marketplace name you run /plugin marketplace remove ifixai-ai first, then the two commands above.

Codex, in your terminal:

codex plugin marketplace add ifixai-ai/iFixAi
codex plugin add ifixai@ifixai-community

Then start Codex and ask "run iFixAi on my setup". Codex asks once to trust the plugin's hook, then provisions the engine on the first session. Already on ifixai@ifixai-ai? codex plugin marketplace upgrade fails on the renamed marketplace, so run codex plugin marketplace remove ifixai-ai first, then the two commands above.

Skill (every agent)

Prefer a single scaffolded file, or use an agent without a plugin? One zero-install command writes a native /ifixai-skill slash command into any agent: Claude Code, Codex, Cursor, VS Code / Copilot, Windsurf, Cline, Continue, Gemini, or Zed (plus an AGENTS.md bridge). Only uv and Python 3.10+ are needed; no API key or provider extra to scaffold:

uvx ifixai install --agents cursor   # any slug: claude, codex, vscode, windsurf, cline, continue, gemini, zed
uvx ifixai install --agents all      # scaffold every agent at once
uvx ifixai install --list            # every supported agent and where its file lands

Then run /ifixai-skill in that agent. It reads your setup, builds the fixture, shows the cost via a free --dry-run, and runs only after you say yes (the run is zero-install too, driving uvx --from "ifixai[]" ifixai run). On a new project, name the agent with --agents (auto-detect only finds agents whose folder already exists). Already have the CLI on your PATH? Drop the uvx prefix. The command is named ifixai-skill so it never collides with the Claude Code plugin's /ifixai; pass --name ifixai for the bare name.

Explicit flags

## 1. Install the CLI + the extra for the provider you'll test
pip install "ifixai[anthropic]"

## 2. Prove the pipeline runs: built-in mock, no keys, no network, ~1s.
##    Expect a FAILING scorecard (15/50) — the bundled default fixture ships
##    seeded defects on purpose so you see what failures look like.
##    Defect map: ifixai/fixtures/default/README.md
ifixai run --provider mock --api-key not-used --eval-mode self

## 3. Get a citable grade: your model graded by a *different* vendor's judge.
##    Pass --fixture : without it the seeded-defect default
##    is used and its failures land on YOUR scorecard.
pip install "ifixai[anthropic,openai]"     # SUT's + judge's SDKs (or ifixai[all])
export ANTHROPIC_API_KEY=sk-ant-...         # the SUT, graded
export OPENAI_API_KEY=sk-...                # the judge, auto-paired from the environment
ifixai run --provider anthropic --api-k