Open sourceNode / CLI

promptfoo

The quickest way to test a prompt across models like ChatGPT and Gemini, from your terminal.

What it is

promptfoo is a free command-line tool. You write your prompts and a few test questions in a small config file, and it runs them against one or more AI models, then scores the answers. It also has a built-in red-team mode that tries to break your app with unsafe or tricky inputs.

Why a QA engineer cares

It feels like the test runners you already know. You get a table of pass and fail, you can compare two models side by side (say ChatGPT vs Gemini), and it drops straight into CI so a prompt change cannot slip through untested.

Get started

Install it, then run the example below.

Install

npx promptfoo@latest init

Quickstart

# promptfooconfig.yaml
prompts:
  - "Answer the question clearly: {{question}}"
providers:
  - openai:gpt-4o
  - google:gemini-1.5-pro
tests:
  - vars:
      question: "What is the capital of France?"
    assert:
      - type: contains
        value: "Paris"
      - type: llm-rubric        # an AI grades the answer
        value: "Factually correct and concise"

# run it, then open the report
npx promptfoo eval
npx promptfoo view

What you get

✓Compare many models and prompts side by side
✓Checks: contains, similar meaning, valid JSON, and AI-graded rubric
✓Built-in red-teaming for prompt injection and unsafe answers
✓Runs in CI and fails the build on a regression

Best for

Prompt testingModel comparisonRed-teamingCI gates

Where it fits

Great as your first tool. Move to DeepEval or Ragas when you need deeper Python metrics.

Official docs for promptfoo →

Other AI-testing tools

DeepEval

Pytest-style AI evals (correct, relevant, bias)

Ragas

Check RAG bots: did they find and use the right source

LangSmith

Trace, save datasets and run evals for AI apps

Giskard

Scan AI models for weak spots and unsafe answers

Confident AI

Dashboards to track eval scores over time

Put it to work

See where promptfoo fits in the full picture, or follow the step-by-step AI for QA roadmap.

The AI testing guide The AI for QA roadmap