PandaProbe is an open source agent engineering platform that lets teams collaboratively trace, evaluate, monitor, and debug AI agents, and it is aimed at teams building agentic applications on frameworks such as LangGraph, CrewAI, Claude Agent SDK, and the OpenAI Agents SDK.

What it is

PandaProbe is a Python project licensed under Apache-2.0 and built by the Chirpz AI team. It lives in the agentic AI ecosystem as an observability and evaluation layer: the FastAPI application server, a Next.js dashboard, Celery worker and beat scheduler, PostgreSQL 16, and Redis 7 combine into a service that ingests agent traces, stores them as traces and spans grouped into sessions, and runs evaluations against them. Teams can consume it two ways — the managed PandaProbe Cloud deployment, or a self-hosted instance run from the repository.

The concrete problem it solves is that agent behaviour is hard to inspect after the fact. A developer working with LangGraph, CrewAI, Claude Agent SDK, or the OpenAI Agents SDK has no built-in place to look at what spans were produced, which sessions a run belongs to, or how a response scored. PandaProbe replaces ad-hoc logging and scattered debugging with a dedicated platform: clients post traces over HTTP, the API enqueues ingestion work, a Celery worker persists trace and span records into PostgreSQL, and evaluations run as queued jobs whose verdicts and scores are also persisted.

Key capabilities

• Trace ingestion over HTTP with X-API-Key and X-Project-Name headers on POST /traces, returning 202 Accepted after the job is enqueued to Redis.
• Trace and session retrieval through GET /traces and GET /sessions, with filtering and paginated responses queried from PostgreSQL.
• Agent evaluation through POST /evaluations, which enqueues an evaluation job that a Celery worker executes as an LLM-as-a-judge call via LiteLLM, persisting the verdict and score.
• Management plane using Authorization: Bearer tokens verified against an identity provider (Supabase or Firebase), exposing /user, /organizations, and /projects.
• Client integrations for LangGraph, CrewAI, Claude Agent SDK, and OpenAI Agents SDK.
• Next.js dashboard on port 3000 and a Scalar-generated API reference at http://localhost:8000/scalar.
• Self-hosted service set covering frontend, app, worker, beat, postgres, and redis, driven by ./start.sh.

Who uses it and how

• Teams collaboratively debugging agents, since the platform is explicitly built around shared tracing, evaluation, and monitoring rather than single-developer log files.
• Self-hosters who need the full service to run inside their own environment, using the Docker-based ./start.sh path and their own PostgreSQL, Redis, and identity provider.
• Developers already building on LangGraph, CrewAI, Claude Agent SDK, or OpenAI Agents SDK, who add tracing by pointing a client library at the API.
• Evaluation-driven workflows, where LLM-as-a-judge scoring is run as background jobs over stored traces rather than as inline application code.
• Teams wanting a managed option first, using PandaProbe Cloud's free tier without a credit card before committing to self-hosting.

Getting started

Self-hosting requires Docker installed and running: clone https://github.com/chirpz-ai/pandaprobe.git, enter the directory, and execute ./start.sh, then open the dashboard on port 3000 or the API reference on port 8000. PandaProbe Cloud is the managed alternative, and the client library documentation covers quickstart and advanced integrations.

How it compares

This registry entry names no paid products that PandaProbe replaces, and it names no directly comparable tracing or evaluation tools either; the projects referenced are integration targets, not alternatives. PandaProbe therefore stands alone in this registry as the agent observability and evaluation platform listed here.

When to use it — and when not to

A self-hoster must operate the full stack: PostgreSQL 16, Redis 7, the FastAPI app, the Next.js frontend, and both the Celery worker and beat scheduler, plus an external identity provider such as Supabase or Firebase and an LLM engine for judge-based evaluations. Teams unwilling to run that surface, or to depend on a cloud identity provider for authentication, should not choose the self-hosted path. The README excerpt is also thin — it covers architecture, services, and startup but leaves client usage to external documentation, so anyone needing detailed in-repo guidance should plan around that.

git clone https://github.com/chirpz-ai/pandaprobe.git
cd pandaprobe
./start.sh

sequenceDiagram
    participant Client as 📡 SDK / HTTP Client
    participant API as ⚡ FastAPI
    participant Auth as 🔐 Auth Service
    participant IdP as 🌐 Supabase / Firebase
    participant Identity as 👥 Identity Service
    participant Trace as 🫆 Trace Service
    participant Eval as 🧪 Eval Service
    participant DB as 🗄️ PostgreSQL
    participant Redis as 📮 Redis
    participant Worker as ⚙️ Celery Worker
    participant LLM as 🤖 LLM Engine (LiteLLM)

    Note over Client,API: Management Plane (Bearer token)
    Client->>API: Authorization: Bearer <idp_token>
    API->>Auth: Verify token
    Auth->>IdP: Validate with provider
    IdP-->>Auth: User identity
    Auth-->>API: Authenticated user
    API->>Identity: /user, /organizations, /projects
    Identity->>DB: Read / write
    DB-->>Identity: Result
    Identity-->>Client: Response

    Note over Client,API: Data Plane (API key)
    Client->>API: X-API-Key + X-Project-Name
    API->>Identity: Resolve org & project
    Identity-->>API: Project context

    API->>Trace: POST /traces
    Trace->>Redis: Enqueue ingestion job
    Redis-->>Client: 202 Accepted
    Redis->>Worker: Pick up job
    Worker->>DB: Persist trace + spans

    API->>Trace: GET /traces, /sessions
    Trace->>DB: Query with filters
    DB-->>Trace: Rows
    Trace-->>Client: Paginated response

    API->>Eval: POST /evaluations
    Eval->>Redis: Enqueue eval job
    Redis-->>Client: 202 Accepted
    Redis->>Worker: Pick up job
    Worker->>LLM: LLM-as-a-judge call
    LLM-->>Worker: Verdict + score
    Worker->>DB: Persist evaluation result