Open-source framework for evaluating LLM outputs with unit-test style metrics.
Open-source framework for evaluating LLM outputs with unit-test style metrics.
The LLM Evaluation Framework
Documentation |
Metrics and Features |
Getting Started |
Integrations |
Confident AI
Deutsch |
Español |
français |
日本語 |
한국어 |
Português |
Русский |
中文
DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which use LLM-as-a-judge and other NLP models that run locally on your machine.
Whether you're building AI agents, voice agents, multi-turn chatbots, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:
• LLM apps end-to-end as black boxes
• Complete AI agent trajectories across every decision and action
• Individual AI agent steps such as LLM calls, tool use, retrieval, and sub-agent handoffs
Use these evaluations to determine the optimal models, prompts, and architecture to improve your AI quality, prevent prompt drifting, or even transition from OpenAI to Claude with confidence.
Jev is now available in DeepEval, both in existing metrics and the latest JevEval.
IMPORTANT:
If you're already in production, Confident AI surfaces failing traces before anyone has written an eval for them, clusters them into failure modes, and turns them into your next eval.
🔥 Metrics and Features
• 📐 Large variety of ready-to-use LLM eval metrics (all with explanations) powered by ANY LLM of your choice, statistical methods, or NLP models that run locally on your machine covering all use cases:
• Custom, All-Purpose Metrics:
• JevEval — a custom metric scored by Jev, a System One model that answers your questions with calibrated probabilities instead of generated text
• G-Eval — a research-backed LLM-as-a-judge metric for evaluating on any custom criteria with human-like accuracy
• DAG — DeepEval's graph-based deterministic LLM-as-a-judge metric builder
• Task Completion — evaluate whether an AI agent accomplished its goal
• Tool Correctness — check if the right tools were called with the right arguments
• Goal Accuracy — measure how accurately the agent achieved the intended goal
• Step Efficiency — evaluate whether the agent took unnecessary steps
• Plan Adherence — check if the agent followed the expected plan
• Plan Quality — evaluate the quality of the agent's plan
• Tool Use — measure quality of tool usage
• Argument Correctness — validate tool call arguments
• Agent Responsiveness — check whether the voice agent answered when it was spoken to
• Turn-Taking Naturalness — measure response latency and interruptions across the call
• Voice Naturalness — evaluate how natural the voice agent sounds
• Transcription Accuracy — check whether the voice agent's speech-to-text heard the caller correctly
• Audio Glitch — detect audio artifacts in the voice agent's speech
Multi-Turn Metrics …
Builder tool · Builder tools / Observability & evals · Open Source · Apache-2.0 · Python
Website https://deepeval.com/ · Repo https://github.com/confident-ai/deepeval
19,451 stars · 2,194 forks · 829 open issues · pushed 2026-11-12 · verified 2026-11-12
Tags: ai-agents, tool, observability-evals
What is DeepEval? DeepEval is a coding website listed on Scriptiii Webnest. Open-source framework for evaluating LLM outputs with unit-test style metrics. Pricing: Free. Free to use. Best for: ai-agents, tool, observability-evals.
Everything you need to know about DeepEval
Similar websites in Coding
Open-source vector database that stores objects and vectors with hybrid search and filtering.
Coding·Free