One entrypoint to evaluate everything.
Classic ML models, LLMs, RAG systems, agents and RL checkpoints, offline and online. Pluggable, sandboxed and self-hosted, from a laptop to Kubernetes.
pip install evalsiLatest releaseChangelog
$ uv run evalsi eval --data qa.jsonl --evaluators exact-match,numeric-match,latency
6 records · 3 evaluators
metric n mean 95% CI skipped errors
exact-match 6 0.333 [0.097, 0.700] 0 0
numeric-match 3 0.667 [0.208, 0.939] 3 0
latency 6 517.500 [276.409, 758.591] 0 0Works with the tools you already use
Four doors, one platform
Score it, run it, watch it, reward it
The same evaluators work in every mode, so a metric means the same thing in CI, in production and in training.
evalsi evalScore
Grade outputs you already have. Every mean ships with a confidence interval; records an evaluator can't judge are skipped, never scored zero.
evalsi runRun
Durable, resumable runs with trials, gates and budgets, including agents on sandboxed tasks and benchmarks.
evalsi policyWatch
Continuously evaluate live agent traces, sent over OpenTelemetry or read from MLflow, Phoenix and Langfuse, with sampling, cascades, windowed alerts and promotion to datasets.
RewardServiceReward
Evaluators as RL rewards for TRL, verl and OpenRLHF, and every checkpoint checked against the base model.
Fits your stack
Plug in at either end
Send traces from any OpenTelemetry source or gateway, or read them from the MLflow, Phoenix or Langfuse you already run, point runs at your own agent over A2A, MCP, HTTP or a CLI, and export results to the dashboards you already watch.
- gRPC, gRPC-Web, HTTP/JSON and MCP on one port
- Scores written back to MLflow, Langfuse and Phoenix, or exported over OpenTelemetry
- No default judge, so nothing is billed by surprise
Built for production
Everything an evaluation platform needs
Sandboxed, fail-closed
Firecracker where available, then bubblewrap or Landlock, then hardened pods. If nothing qualifies, nothing runs.
Pluggable by design
Existing frameworks and benchmarks plug in as isolated adapters.
Inline guardrails
Redact, evaluate and block in the request path through agentgateway.
apiVersion: evals.si/v1alpha1
kind: EvalRun
metadata: {name: swebench-verified}
spec:
target: {agent: {cli: {command: [my-agent]}}}
dataset: {uri: "swebench://verified"}
evaluators: [{ref: task-success}]
gates:
- {metric: task-success, min: 0.3}Kubernetes native
Helm charts and an operator with EvalRun, OnlineEvalPolicy, TraceSource, Evaluator and SandboxClass resources.
exact-match 0.93 ▲ 0.04 · gate passed
MCP for coding agents
Agents check their own changes with evalsi mcp.
Identity and access
OIDC, API keys, project RBAC, CEL rules and an audit log.
Runs in your environment
Self-hosted with your models, storage, identity and secrets.
Trace sources
Score traces kept in MLflow, Phoenix or Langfuse, and write scores back
Agent frameworks
LangGraph, CrewAI, OpenAI Agents, LlamaIndex, Claude Agent SDK
Agent harness integration
Online scoring and deploy gates beside your agent harness
Webhooks
Signed events when a run finishes or fails a gate
Human annotation
Queues with progress and agreement
Wasm evaluators
Ship graders as Wasm with a plugin index
Read-only web UI
Runs, comparisons, policies and catalog
Fine-tuning and RL
Rewards and checkpoint regression checks
Your way in
CLI, Python or Kubernetes
Write an evaluator as a decorated function, run it next to the built-ins, then take the same spec to a cluster as an EvalRun that gates your rollout.
# Install from PyPI and grade outputs you already have
pip install evalsi
evalsi eval --data qa.jsonl \
--evaluators exact-match,numeric-match,latency
# Add a judge: any OpenAI-compatible server, or Claude
pip install 'evalsi[anthropic]'
evalsi eval --data qa.jsonl --evaluators llm-judge \
--judge-provider anthropic --judge-model claude-opus-5-5Start evaluating in a minute
Open source, self-hosted and air-gappable. Bring your own models, storage, identity and secrets.