[opensourcedex]

api-relay-audit is a free, open source ai security & privacy project written in Python and released under AGPL-3.0. It has 839 GitHub stars, 80 forks and 26 open issues, and was last pushed 40 hours ago. On this registry it ranks #30 of 34 tracked projects in AI Security & Privacy, with 5 head-to-head comparisons available. It gained 2 stars over the last 3 tracked days.

What is api-relay-audit?

What it is

API Relay Audit is an independent, local security audit tool for AI API relays, mirrors, gateways, and LLM proxies. It is written in Python, released under the AGPL-3.0 license, and lives in the AI security and privacy ecosystem alongside the OpenAI-compatible and Anthropic-compatible proxy tooling it is designed to examine. The project is informed by the paper Your Agent Is Mine, Anthropic's September 10, 2026 threat intelligence report on fraudulent Claude resellers, and a September 11 disclosure by co-author Chaofan Shou regarding router data containing user credentials. It ships both as a standalone script and as a DSH plugin.

The concrete problem is that a user who routes traffic through a third-party relay cannot easily tell what that relay does to the traffic. A relay may inject hidden prompts, swap the requested model, truncate context, rewrite tool-call output, or leak credentials through error responses. API Relay Audit checks observable relay behavior and produces a Markdown report so a caller can evaluate a relay before sending real traffic, coding-agent automation, package-install suggestions, or wallet-related actions. Your API key is sent only to the relay URL you choose.

Key capabilities

  • Detects hidden prompt injection, instruction override, prompt leakage, and extraction behavior.
  • Collects model identity, stream, latency, and upstream channel signals as substitution indicators.
  • Flags changes to pinned package-command text and other tool-call rewriting.
  • Reports error-response leakage and SSE stream anomalies.
  • Provides per-step evidence with LOW, MEDIUM, and HIGH finding summaries.
  • Keeps inconclusive probes visible rather than folding them into a clean result.
  • Offers general, full, and web3 profiles bound to defined query families.

Who uses it and how

  • Users of third-party AI API relays, mirrors, gateways, or LLM proxies checking behavior before production traffic.
  • Teams verifying whether a Claude-compatible or OpenAI-compatible proxy injects prompts, swaps models, truncates context, or rewrites tool output.
  • Web3 and wallet users running --profile web3 before agent workflows touch signing or transactions.
  • Operators who need a local, repeatable audit report instead of a web tool that asks for an API key.

Getting started

Fetch the script with curl at a pinned ref such as AUDIT_SCRIPT_REF=v2.4.0, then run python audit.py --key --url --output report.md; add --profile web3 for wallet-sensitive checks. Use master as the ref only when intentionally testing unreleased changes.

When to use it — and when not to

Use it when a third-party relay sits between your agent and the model provider and you want observable evidence before trusting it. It does not certify that a relay is safe, does not replace manual security review or operational monitoring, and does not treat inconclusive as clean. Blocked probes and ambiguous responses stay

project readme (upstream, from github) — read inline

API Relay Audit

Local security audit for AI API relays and LLM proxies.

GitHub Pages README 中文 X @li9292 GitHub toby-bridges

DSH Plugin

Your Agent Is Mine: what you can test locally

Your Agent Is Mine documents malicious API relays injecting payloads and exfiltrating credentials. Anthropic's September 10, 2026 report describes fraudulent Claude resellers swapping models and harvesting credentials through their client tooling. In his September 11 disclosure, co-author Chaofan Shou reports buying router data containing users' credentials.

API Relay Audit is an independent, local security audit tool for AI API relays and LLM proxies, informed by the paper. The current release checks observable relay behavior and generates a Markdown report covering:

  • Prompt and context signals: hidden prompt injection, instruction override, and context truncation.
  • Response integrity: changes to pinned package-command text, error-response leakage, and SSE stream anomalies.
  • Reviewable findings: per-step evidence and LOW / MEDIUM / HIGH summaries; inconclusive probes remain visible.

See the output: example report (synthetic fixture) · Try it: run a local audit · Coverage and limits

The standalone script uses Python's standard library plus curl. Your API key is sent only to the relay URL you choose.

Quick Start

AUDIT_SCRIPT_REF=v2.4.0
curl -fsSL "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" -o audit.py

python audit.py --key <YOUR_KEY> --url <BASE_URL> --output report.md

# Web3 / wallet users
python audit.py --key <YOUR_KEY> --url <BASE_URL> --profile web3 --output report.md

See a public-safe fixture report: sanitized audit report. Use master as AUDIT_SCRIPT_REF only when intentionally testing unreleased changes.

If API Relay Audit helps you evaluate a relay before sending real traffic, star the repository to follow new detector coverage and release-tested updates.

When to Use It

  • You use a third-party AI API relay, mirror, gateway, or LLM proxy.
  • You want to check whether a Claude-compatible or OpenAI-compatible proxy injects prompts, swaps models, truncates context, or rewrites tool output.
  • You are testing relay behavior before production traffic, coding-agent automation, package-install suggestions, or wallet-related actions.
  • You need a local, repeatable audit report instead of a web tool that asks for your API key.

What It Does Not Claim

  • It does not certify that a relay is safe.
  • It does not replace manual security review or operational monitoring.
  • It does not treat inconclusive as clean; blocked probes and ambiguous responses stay visible in the report.

Query Family Boundaries

Query family User intent Profile / steps Evidence boundary
API relay audit Audit a third-party relay, mirror, gateway, LLM proxy, or resale API before trusting traffic. general by default; full for every probe Produces a local report, not a safety certificate.
Prompt injection audit Detect hidden prompt injection, prompt leakage, instruction override, and extraction behavior. general; Steps 3-6 Records prompt evidence without publishing private prompts or secrets.
Model substitution signals Collect model identity, stream, latency, and upstream channel signals. general; Steps 5, 10, 13, 14 Self-ID, latency, and channel fingerprints are signals, not standalone proof of provider substitution.
Web3 relay audit Check wallet-sensitive relay behavior before agent workflows touch signing or transactions. web3 or full; Step 11 Profile-gated; general relay audits do not imply wallet safety.

The canonical contract lives in docs/query-families.md. README headings, Pages cards, issue templates, and skill descriptions should preserve these boundaries instead of flattening them into one slogan.

Coverage

API Relay Audit checks whether a relay modifies the request or response path between you and the model:

  • Prompt safety: token injection, prompt extraction, instruction override, jailbreak resistance
  • Relay integrity: context truncation, tool-call substitution, error leakage, stream integrity
  • Model identity: non-Claude identity leaks, model substitution signals, Claude/OpenAI-compatible relay behavior
  • Web3 wallet safety: transfer guidance, signed-transaction refusal, private-key refusal

Audit LLM Proxies Locally

The project has two distribution modes:

  • audit.py: zero-dependency standalone script for quick local audits
  • api_relay_audit/ plus scripts/: modular development version with tests

Runtime profiles:

  • general: default AI API relay and LLM proxy checks
  • web3: wallet-safety probes for Web3 agent flows
  • full: general plus Web3 checks

DeepSeek Harness DSH Plugin

The repository is also an installable dsh-api-relay-audit bundle for DeepSeek Harness Web and community TUI surfaces that use the official @deepseek-ai/dsh-commands registry. Pin an immutable commit or release tag:

DSH_PLUGIN_REF=v2.4.0
dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"

# dsh-cc-tui and other compatible profile-based clients
dsh plugin --profile cc-tui add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"

The command reuses the current DSH provider's baseURL, model, and credential reference. The credential stays in DSH Credentials and is delivered to the local audit process through an environment variable, never through command arguments or the session log:

/relay-audit
/relay-audit --connectivity
/relay-audit --profile web3 --fast-context
/relay-audit --url <URL> --model <claude-model> --credential-ref <DSH_CREDENTIAL_REF>

No arguments preserves the existing full-audit default and may consume metered tokens. Use --connectivity for a lower-cost check. This distribution does not add a new model baseline: the selected route must identify as Claude, although the relay API itself may be Anthropic-compatible or OpenAI-compatible. Independent wrappers without DSH profiles and the DSH command registry are not compatible with this bundle. See agent distribution notes. The exact v2.4.0 installation, runtime, and secret-scan results are recorded in the DSH distribution verification.

Retained Agent Skill Files

The repository retains its existing OpenClaw and Hermes skill files for direct users and downstream compatibility. They are not current registry distribution targets; active distribution and release verification focus on the DeepSeek Harness plugin.

  • OpenClaw Skill: run a local AI API relay audit before an OpenClaw agent depends on a third-party relay, proxy API, or resale key.
  • Hermes Skill: install API Relay Audit as a Hermes Agent skill and run the same local 14-step LLM proxy security audit from an agent workflow.

These files do not certify that a relay is safe. They help agents generate a local, reviewable Markdown report before trusting a relay path.

Evidence Boundaries

Natural-language self-identification is treated as a consistency signal, not upstream proof. A response saying it is Qwen, DeepSeek, GPT, or Claude can indicate a mismatch, but it does not by itself prove that a provider substituted the upstream model.

Stronger claims require corroborating evidence such as raw response JSON, request IDs, provider/model metadata, stream signatures, transparent-log hashes, and reproducible runs. Public submissions should use r

readme truncated — read the full docs on github

Frequently asked questions

Is api-relay-audit free to use?

api-relay-audit is open source under the AGPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does api-relay-audit do?

Local security audit for AI API relays and LLM proxies: detects prompt injection, model substitution, tool-call rewriting, SSE anomalies, error leakage, and Web

What is api-relay-audit written in?

api-relay-audit is primarily written in Python. Its source is publicly available at https://github.com/toby-bridges/api-relay-audit, and it has 839 GitHub stars.