Open sourcePython

Ragas

The go-to tool for testing a RAG bot, the kind that answers from your own documents.

What it is

Ragas is a free Python library made just for RAG apps (a bot that searches your documents, then answers from them). It scores both halves of the job: did it fetch the right document, and did the final answer stick to that document.

Why a QA engineer cares

A RAG bot can give a nice answer that is not actually in your docs. Ragas catches that. It separates a retrieval problem (wrong document found) from an answer problem (right document, made-up answer), so you know what to fix.

Get started

Install it, then run the example below.

Install

pip install ragas

Quickstart

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset

data = {
    "question": ["What is the refund window?"],
    "answer": ["You can get a refund within 7 days."],
    "contexts": [["Refunds are given within 7 days of purchase."]],
    "ground_truth": ["7 days"],
}
result = evaluate(Dataset.from_dict(data),
                  metrics=[faithfulness, answer_relevancy, context_precision])
print(result)

What you get

✓Retrieval scores: context precision and context recall
✓Answer scores: faithfulness and answer relevancy
✓Can build a test set from your own documents
✓Works with LangChain, LlamaIndex and plain Python

Best for

RAG evaluationRetrieval qualityGrounding checks

Where it fits

Use with DeepEval or promptfoo, which handle general answer quality and safety.

Official docs for Ragas →

Other AI-testing tools

promptfoo

Run evals and safety tests on prompts and models, works in CI

DeepEval

Pytest-style AI evals (correct, relevant, bias)

LangSmith

Trace, save datasets and run evals for AI apps

Giskard

Scan AI models for weak spots and unsafe answers

Confident AI

Dashboards to track eval scores over time

Put it to work

See where Ragas fits in the full picture, or follow the step-by-step AI for QA roadmap.

The AI testing guide The AI for QA roadmap