Open sourcePython

DeepEval

Write AI tests the way you write Pytest, with ready-made scores for quality and safety.

What it is

DeepEval is a free Python library for testing AI answers. You describe a test case (the input, the answer your app gave, and the source it should use), then attach metrics like Answer Relevancy, Faithfulness, Hallucination and Bias. Each metric gives a score from 0 to 1, and the test passes if it clears your threshold.

Why a QA engineer cares

If you know Pytest, you already know DeepEval. Same files, same `assert`, same `deepeval test run`. It turns fuzzy questions like 'is this answer good?' into numbers you can gate on.

Get started

Install it, then run the example below.

Install

pip install deepeval

Quickstart

# test_llm.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

def test_refund_answer():
    case = LLMTestCase(
        input="What is our refund policy?",
        actual_output=my_app("What is our refund policy?"),
        retrieval_context=["Refunds are given within 7 days of purchase."],
    )
    assert_test(case, [
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.7),
    ])

# run
# deepeval test run test_llm.py

What you get

✓14+ ready metrics: relevancy, faithfulness, hallucination, bias, toxicity
✓GEval: write your own scoring rule in plain English
✓Pytest integration, so it fits your existing test suite
✓Uploads results to the Confident AI dashboard if you want

Best for

RAG and chatbot testingRegression suitesCustom scoring

Where it fits

Confident AI is its hosted dashboard. Use Ragas alongside it for deeper RAG retrieval scores.

Official docs for DeepEval →

Other AI-testing tools

promptfoo

Run evals and safety tests on prompts and models, works in CI

Ragas

Check RAG bots: did they find and use the right source

LangSmith

Trace, save datasets and run evals for AI apps

Giskard

Scan AI models for weak spots and unsafe answers

Confident AI

Dashboards to track eval scores over time

Put it to work

See where DeepEval fits in the full picture, or follow the step-by-step AI for QA roadmap.

The AI testing guide The AI for QA roadmap