AWS Details UAEF for Cross-Framework AI Agent Evaluation

AWS on November 11, 2026, published a technical introduction to the Universal Agent Evaluation Framework (UAEF), an open-source system licensed under Apache-2.0 that evaluates AI agents using 36 built-in metrics across seven dimensions.

UAEF is designed to keep evaluations comparable across LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore. Those frameworks record messages, tool calls and agent-to-agent handoffs differently, so evaluation code tied to one format can require new metrics, scoring logic and result storage after a framework change. That makes it harder to determine whether a change to the prompt, model or tools improved the agent.

AWS reports that customers across 12 organizations in telecom, healthcare and financial services used UAEF to apply consistent metrics and compare agent versions before release. The awslabs/uaef repository contains the Python library, framework adapters, metrics, AWS deployment infrastructure, web interface and runnable notebooks.

Canonical Traces Enable Shared Scoring

Adapters normalize each agent’s messages, tool calls and handoffs into a canonical AgentTrace, the only representation the scoring engine uses. UAEF can adapt an existing trace from a past run, log or observability store, or invoke an agent through a runtime Amazon Resource Name or HTTP endpoint and convert the resulting execution. A generic JSON adapter extends the process to custom frameworks.

The metrics catalog groups the built-ins into tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning. Scores use a normalized 0-to-1 scale, where 1 is best. Deterministic metrics calculate directly from traces and ground truth, while LLM-as-a-judge metrics use a model on Amazon Bedrock for qualities such as relevance and safety. Custom metrics and RAGAS and DeepEval integrations are also supported.

Ground-truth answers are optional overall, although reference-comparison metrics require them. Tool-calling evaluation can also define expected tools, their order and their arguments. When a metric lacks necessary input, it returns no score with a warning, and that empty result is excluded from dimension averages rather than counted as zero. Partially labeled datasets and unlabeled production traffic can therefore still be evaluated with metrics whose inputs are available.

Stored Experiments Expose Regressions

Persisted evaluations become experiments. Amazon DynamoDB stores metadata for listing and comparison, including evaluation counts and average scores, while Amazon S3 holds full JSON results with per-metric scores and judge reasoning. Individual runs can record the framework, model, tools, system prompt, parameters, configuration snapshot and Git commit, tying each score to the agent design under test. UAEF screens traces for personal information before storage, and redaction is available for stored results.

For metrics shared by two runs, the comparison engine calculates absolute and percentage changes against a baseline. A configurable threshold, set to 5% by default, separates improvements, regressions and unchanged results; regressions are graded minor, moderate or severe and grouped by dimension. That grouping can show whether a prompt revision affected tool calling without changing response quality.

Reporting keeps reproducible statistics separate from model-generated interpretation. The web interface shows per-test-case and per-dimension results, and an optional statistical report covers score distributions, confidence intervals, outliers, metric correlations and failed test cases without using an LLM. A separate Bedrock-generated analysis can propose root causes and prioritized next steps for one run or a comparison.

The Python library and low-code web service use the same evaluation engine and stores. Teams can run evaluations in notebooks, scripts and continuous integration and continuous delivery pipelines or through a browser without changing how results are scored and compared.

AWS Dependencies and Isolation Boundaries

The installation guide requires Python 3.10 or later and the uv package manager. AWS credentials are needed for Bedrock-based judging and DynamoDB/S3 persistence, so teams using those judge and storage features remain connected to AWS services even when their agents use different frameworks.

UAEF treats evaluated-agent output as untrusted input to judge prompts. It sends the rubric separately, escapes the content, strictly parses judge responses and requires a verbatim evidence span before accepting a score. AWS says judge scores should not gate security controls; runtime enforcement belongs in a separate Amazon Bedrock Guardrails invocation path.

The repository describes UAEF as a reference architecture designed to run inside a platform that supplies user and tenant isolation. Direct multi-tenant deployment without that host enforcement is unsupported. Each tenant needs a separate deployment with its own AWS account or, at minimum, its own stack, DynamoDB tables, S3 buckets and Cognito user pool.

For enterprises running several agent frameworks, UAEF offers a stable scoring baseline that can survive changes in prompts, models, tools or implementation. Realizing that continuity requires more than installing the library: useful comparisons depend on suitable metrics and datasets, while multi-tenant use depends on isolation supplied outside UAEF.