Get started

Install, grade your first outputs, run evaluators from Python or as a server, then take a tour of runs, agents and online evaluation.

Install

Each release publishes (see the Kubernetes guide to verify the signatures):

• Server and CLI binaries: Linux amd64 and arm64 archives on the releases page.
• Python: pip install evalsi (SDK and CLI) and pip install evalsi-harness (the agent harness) from PyPI; the same wheels and sdists are attached to the release.
• Kubernetes: multi-architecture images at ghcr.io/abhishek-rnjn/evalsi and the three Helm charts as OCI artifacts, oci://ghcr.io/abhishek-rnjn/charts/evalsi, evalsi-crds and evalsi-sandboxd.

To try it from source, use the quickstart below.

Quickstart

You need uv (it installs Python 3.11+ for you). Running the server also needs Go 1.26+. On Linux, the sandbox for code evaluators needs bubblewrap (see Development).

git clone https://github.com/Abhishek-Rnjn/Evals.si && cd Evals.si
cd python && uv sync --all-packages
uv run evalsi eval --data ../examples/quickstart/qa.jsonl --evaluators exact-match,numeric-match,latency

6 records · 3 evaluators

metric         n  mean     95% CI              skipped  errors
exact-match    6  0.333    [0.097, 0.700]            0       0
numeric-match  3  0.667    [0.208, 0.939]            3       0
latency        6  517.500  [276.409, 758.591]        0       0

Records that lack what an evaluator needs (here, a numeric reference) are skipped, never scored as zero. Failures in the evaluator or judge are reported as errors and excluded from the metrics. Use --cluster-by <metadata key> when records are correlated.

Add a judge

There is no default judge. Configure one explicitly:

# Any OpenAI-compatible server: vLLM, Ollama, LiteLLM, an AI gateway, OpenAI itself
uv run evalsi eval --data ../examples/quickstart/qa.jsonl --evaluators exact-match,llm-judge \
  --judge-model my-judge --judge-base-url http://localhost:8000/v1

# Claude, through the official SDK (installed by the `anthropic` extra)
uv run evalsi eval --data ../examples/quickstart/qa.jsonl --evaluators llm-judge \
  --judge-provider anthropic --judge-model claude-opus-5-5 --param llm-judge.rubric=helpfulness

Judge responses are cached by content hash, so re-running an unchanged evaluation costs nothing. Pass --no-cache to force fresh calls.

From Python

import evalsi
from evalsi import Record, Score, evaluator

@evaluator(name="acme/polite", version="1.0.0")
def polite(record: Record) -> Score:
    return Score(passed="please" in record.output.as_text().lower())

result = evalsi.evaluate("qa.jsonl", ["exact-match", polite])
print(result.table())
result.save("results.json")   # manifest, summaries and every per-record result

evalsi catalog lists every installed evaluator and its params.

As a server

evalsid serves the same evaluators over gRPC, gRPC-Web, HTTP/JSON and REST on one port, and runs the Python evaluators in a supervised worker.

go build -o bin/evalsid ./cmd/evalsid        # from the repository root
cd python && uv sync --all-packages
EVALSID=../bin/evalsid uv run evalsi serve --config ../examples/server/evalsi.yaml

from evalsi.client import Client

with Client("http://localhost:8080") as client:          # API key from EVALSI_API_KEY
    print(client.evaluate("qa.jsonl", ["exact-match"]).table())

Client.evaluate() returns the same result as in-process evalsi.evaluate(), intervals included; evaluate_async() and evaluate_stream() (with evalsi[grpc]) don’t block. REST lives under /v1alpha1 (POST /v1alpha1/evaluate, POST /v1alpha1/runs, GET /v1alpha1/traces/{trace_id}, GET /v1alpha1/scores for online scores per trace and label, …; the full list is in internal/server/rest.go), and gRPC reflection is on.

On loopback with no auth section the server runs without authentication; on any other address it requires it unless you pass --no-auth. The example’s judge is Claude: export ANTHROPIC_API_KEY first, or llm-judge reports errors (never zeros). To try API keys, roles and the audit log locally, use examples/auth/local.yaml and the identity guide.

A tour

Runs and CI gates

A run spec generates outputs from a target, scores them and checks gates. The same file runs embedded, on a server, or as a Kubernetes EvalRun:

uv run evalsi run -f ../examples/runs/capitals.yaml                               # embedded
uv run evalsi run -f ../examples/runs/capitals.yaml --server http://localhost:8080
uv run evalsi compare --server http://localhost:8080 <baseline-run> <candidate-run>

trials: 3 adds pass@3 and pass^3, gates fail the command with exit code 3 (so CI fails), and budgets cap tokens. Server runs are durable: a run interrupted by a restart resumes without redoing finished work. A project can subscribe to signed webhooks for finished runs, failed gates, scored traces and alerts. See examples/ci for a pull-request gate.

Agent runs on sandboxed tasks

Each task gets its own environment: an OCI image, files, setup and a sandbox policy. The agent works in it, a checker grades the end state, and evaluators score the trajectory and the diff.

uv run evalsi run -f ../examples/agents/swebench-verified.yaml --server http://localhost:8080
uv run evalsi promote <run> --dataset regressions --when 'scores["task-success"] < 1' --server http://localhost:8080

swebench://, taubench://, harbor://, terminal-bench:// and bfcl:// datasets are graded by each benchmark’s own code. The harness adds MCP tools, mocks and fault injection, a simulated user, and record and replay. See the agent runs guide.

Online evaluation of live agents

Send traces to the server’s OTLP endpoint, or read them from where they already are, then apply a policy: a CEL selector, deterministic sampling, cascades (the judge runs only after cheap checks pass), windowed alerts, and promotion of bad traces into datasets.

uv run evalsi policy apply -f ../examples/watch/support-policy.yaml --server http://localhost:8080
uv run evalsi source apply -f ../examples/sources/studio-mlflow.yaml --server http://localhost:8080

Framework	Tested instrumentations
LangGraph / LangChain	OpenInference, LangSmith’s OTel export, MLflow autolog
CrewAI	MLflow autolog, OpenInference, OpenLLMetry
OpenAI Agents SDK	OpenInference, OpenLLMetry
LlamaIndex	OpenInference
Claude Agent SDK	OpenInference, Claude Code’s own traces

Each row is a real run of the framework, checked in and tested; the agent frameworks guide has the setup and what each instrumentation records. Trace sources read MLflow, Phoenix or Langfuse (history included), score the traces, and write the scores back as MLflow assessments, Phoenix annotations or Langfuse scores. Self-hosted Langfuse 4.56.0 passed the recorded two-stage check; Databricks numeric-experiment bearer-token downloads are built and contract-tested, not live-verified. To add Evals.si to an agent platform, follow the integration guide and the reference demos, which evaluate Deep Agents and DeepSeek Harness agents end to end.

Evaluators and adapters

Pack	Evaluators
core	exact, contains, fuzzy, numeric and regex match; JSON validity and schema; length; latency; token usage; cost
judge	llm-judge with rubrics
text	BLEU, corpus BLEU, ROUGE-1/2/L, chrF, token F1
rag	faithfulness, answer relevance, context precision and recall, citation accuracy
safety	PII, secret and canary leaks; refusal; harmlessness
agent	tool-call accuracy, trajectory match, tool errors, loop detection, step budget, goal completion; task success (pass@k, pass^k), code quality, policy violations, efficiency
code	unit tests run in the sandbox, Python syntax
rl	verifiers for RL rewards: format, math answers, sandboxed code tests, overlong penalty, reward models
finetune	diversity, calibration, contamination, reward hacking
ml-classic	classification, regression and ranking metrics
ml-monitoring	drift (PSI, KS, Jensen-Shannon), data quality, group fairness

DeepEval, RAGAS, Inspect AI and lm-evaluation-harness, and the SWE-bench, τ-bench and BFCL benchmarks, live in python/adapters, each in its own pinned environment. Community evaluators can be Wasm modules with no filesystem, network or clock, installed from a plugin index.

Kubernetes

kubectl create namespace evalsi
helm install evalsi-crds deploy/helm/evalsi-crds --set operator.namespace=evalsi
helm install evalsi deploy/helm/evalsi -n evalsi
kubectl apply -n evalsi -f examples/runs/capitals.yaml
kubectl get evalruns -n evalsi      # PHASE, RUN, DONE, TOTAL

An operator with EvalRun, OnlineEvalPolicy, TraceSource, Evaluator and SandboxClass resources, validated on admission. Also included:

• PostgreSQL, ClickHouse and S3 storage;
• NATS work queues with KEDA scaling;
• HA replicas, sandbox pools, service-account identity;
• an air-gapped bundle;
• a namespace-only install for clusters without cluster-admin.

See the Kubernetes guide.

And more

Identity and access	OIDC (Keycloak, Entra ID, Okta, …), API keys, GitHub Actions OIDC, client certificates; project RBAC with custom roles, CEL rules and an audit log
Inline guardrails	Redact, evaluate and block LLM and MCP traffic in flight through agentgateway, with an audit mode
Trace redaction	Mask credential keys on push and pull before storage and evaluation; configurable key globs, regexes and opt-in PII classes
MCP for coding agents	evalsi mcp lets an agent run a suite before and after its change and see what regressed
Human annotation	Queues with rubrics, annotator agreement, and judge calibration against human labels
Fine-tuning and RL	Evaluators as rewards for TRL, verl and OpenRLHF; every checkpoint checked against the base model
Web UI	Runs, comparisons, policies, annotation queues, guardrails and the catalog, read through the API
Reports and analytics	evalsi report (HTML or Markdown), evalsi analyze (DuckDB over runs), sinks to MLflow, OpenTelemetry, Langfuse and Phoenix, a Grafana dashboard