Examples

Every example lives under examples/ in the repository and is shown here in full.

• Quickstart: grade existing outputs: Score a JSONL file of question/answer records with built-in evaluators.
• Runs: execute a target on a dataset: A durable run with a dataset, evaluators and a pass/fail gate.
• Agent runs on sandboxed tasks: Put the built-in agent, or your own, to work on tasks in a sandbox, including SWE-bench.
• Watch: evaluate live traces: An online policy that scores OpenTelemetry traces as they arrive.
• Trace sources: score traces kept in MLflow: A TraceSource that reads an agent harness’s MLflow traces, scores them and writes the scores back.
• Inline guardrails: Block or flag model output through agentgateway before it reaches users.
• Human annotation queues: Collect human labels and measure agreement alongside automated scores.
• Fine-tuning and RL rewards: Evaluators as RL rewards, and checkpoint checks for forgetting and safety regressions.
• Custom Python evaluator: Write your own evaluator as a decorated function, test it, and install it as a pack so it runs by name.
• Wasm evaluator plugin: Write an evaluator in Go, compile it to Wasm and publish it through the plugin index.
• Identity and access control: Server config with OIDC, API keys and project-scoped roles.
• Server configuration: A baseline evalsid configuration.
• Gate pull requests in CI: A GitHub Actions workflow that fails a build when evaluation scores regress.