Skip to content
Evals.siEvals.si

Custom Python evaluator

Write your own evaluator as a decorated function, test it, and install it as a pack so it runs by name.

Source: examples/custom-evaluator. Read the guide for the concepts behind it.

data.jsonl

Section titled “data.jsonl”
{"id": "good", "input": "What is the capital of France?", "output": "Paris is the capital of France [1]. It lies on the Seine [2].", "context": ["Paris is the capital and largest city of France.", "The Seine flows through Paris."]}
{"id": "dangling", "input": "Who wrote Hamlet?", "output": "Shakespeare wrote Hamlet [3].", "context": ["Hamlet is a tragedy by William Shakespeare."]}
{"id": "uncited", "input": "How tall is Everest?", "output": "Mount Everest is 8,849 metres tall.", "context": ["Mount Everest's height is 8,849 m."]}
{"id": "no-context", "input": "Say hi", "output": "Hi [1]"}

pyproject.toml

Section titled “pyproject.toml”
[project]
name = "acme-evals"
version = "0.1.0"
description = "Example custom Evals.si evaluator pack"
requires-python = ">=3.11"
dependencies = ["evalsi"]


# Registers the pack so `evalsi eval --evaluators acme/valid-citations` finds it.
[project.entry-points."evalsi.packs"]
acme = "valid_citations:PACK"


[build-system]
requires = ["hatchling>=1.25"]
build-backend = "hatchling.build"


[tool.hatch.build.targets.wheel]
include = ["valid_citations.py"]

README.md

Section titled “README.md”
# Custom evaluator


A worked example of a Python evaluator you write yourself: `acme/valid-citations`
checks that a RAG answer's `[n]` citations each point at a retrieved passage.


| File | What it is |
|---|---|
| [`valid_citations.py`](valid_citations.py) | The evaluator: a decorated function |
| [`test_valid_citations.py`](test_valid_citations.py) | Unit tests and an end-to-end run |
| [`data.jsonl`](data.jsonl) | Four records: good, dangling citation, uncited, no context |
| [`run.py`](run.py) | Runs it and prints the results |
| [`pyproject.toml`](pyproject.toml) | Packages it and registers the pack by entry point |


```bash
cd python/evalsi
uv run pytest ../../examples/custom-evaluator              # the tests
PYTHONPATH=../../examples/custom-evaluator uv run python ../../examples/custom-evaluator/run.py
```


```
good        SCORED   ['2 valid citation(s)']
dangling    SCORED   ['cites [3], but only 1 passage(s) were retrieved']
uncited     SCORED   ['0 citation(s), expected at least 1']
no-context  SKIPPED  []


valid-citations: 0.33 over 3 scored, 1 skipped
```


## How it works


```python
@evaluator(
    name="acme/valid-citations",  # namespaced: <owner>/<name>
    version="1.0.0",
    requires=Requirements(output=True, context=True),
    outputs=[MetricSpec("valid-citations", ScoreType.PASSED, higher_is_better=True)],
)
def valid_citations(record: Record, *, min_citations: int = 1) -> Score: ...
```


- **Params** are the keyword-only arguments. `min_citations` defaults to 1;
  pass `valid_citations.bind({"min_citations": 2})`, or `params=` to `evaluate()`.
  Unknown params are rejected.
- **`requires`** declares what a record must have. A record with no `context`
  is **skipped**, not scored as zero, so it does not drag the mean down (the
  `no-context` row above).
- **`SkipRecord`** lets the function skip a record for other reasons. Any
  other exception is reported as an **error** and excluded from the metrics.
- **`Score`** holds exactly one of `number`, `passed`, `label` or `structured`,
  plus an `explanation` that appears in reports.
- The decorated object is callable and `.bind(params).run(...)` runs it the
  way the runner does, which is what the unit tests use.


## Using it elsewhere


- **Python**: pass the object to `evalsi.evaluate(data, [valid_citations])`.
- **By name** (`evalsi eval --evaluators acme/valid-citations`, servers,
  online policies): this example is already a package. `valid_citations.py`
  ends with a `Pack`, and `pyproject.toml` registers it under the
  `evalsi.packs` entry-point group. Install it next to evalsi:


  ```bash
  pip install ./examples/custom-evaluator
  evalsi catalog --pack acme
  evalsi eval --data examples/custom-evaluator/data.jsonl --evaluators acme/valid-citations \
    --param valid-citations.min_citations=2
  ```


- **Sandboxed and shareable**: write it as a Wasm plugin instead, see
  [`../wasm/text-checks`](../wasm/text-checks) and the
  [plugins guide](../../docs/guides/plugins.md). Wasm evaluators cannot call a judge.
- **LLM-judged metric**: add a `ctx` parameter to receive the judge client
  and set `judge=True` in `Requirements`.

run.py

Section titled “run.py”
"""Run the custom evaluator on a dataset.


cd python/evalsi && uv run python ../../examples/custom-evaluator/run.py
"""


from pathlib import Path


from valid_citations import valid_citations


import evalsi


result = evalsi.evaluate(Path(__file__).with_name("data.jsonl"), [valid_citations])


for r in result.results:
    print(f"{r.record_id:<11} {r.outcome.name:<8} {[s.explanation or s.value for s in r.scores]}")
m = result.metric("valid-citations")
print(f"\nvalid-citations: {m.mean:.2f} over {m.n} scored, {m.skipped} skipped")

test_valid_citations.py

Section titled “test_valid_citations.py”
"""Unit tests and an end-to-end check for the custom evaluator.


Run from python/evalsi:  uv run pytest ../../examples/custom-evaluator
"""


from __future__ import annotations


import asyncio
from pathlib import Path


import pytest
from valid_citations import valid_citations


import evalsi
from evalsi import Content, EvalContext, Record, Score


DATA = Path(__file__).with_name("data.jsonl")




def make(output: str, context: list[str] | None = None) -> Record:
    return Record(
        id="r1",
        output=Content.from_value(output),
        context=[Content.from_value(c) for c in context or []],
    )




def score(record: Record, **params: object) -> Score:
    bound = valid_citations.bind(params)
    return asyncio.run(bound.run(record, EvalContext()))[0]




def test_spec_is_declared() -> None:
    spec = valid_citations.spec
    assert spec.ref == "acme/valid-citations@1.0.0"
    assert spec.params == {"min_citations": 1}
    assert spec.requires.context




@pytest.mark.parametrize(
    ("output", "context", "params", "passed"),
    [
        ("Paris [1].", ["a"], {}, True),
        ("A [1] and B [2].", ["a", "b"], {}, True),
        ("A [1] and B [2].", ["a"], {}, False),  # [2] has no passage
        ("A [0].", ["a"], {}, False),  # markers are 1-based
        ("No citations.", ["a"], {}, False),
        ("No citations.", ["a"], {"min_citations": 0}, True),
        ("Only one [1].", ["a", "b"], {"min_citations": 2}, False),
    ],
)
def test_scoring(output: str, context: list[str], params: dict[str, int], passed: bool) -> None:
    result = score(make(output, context), **params)
    assert result.passed is passed
    assert result.name == "valid-citations"
    assert result.explanation




def test_dangling_citation_is_named_in_the_explanation() -> None:
    assert "[3]" in score(make("X [3].", ["a"])).explanation




def test_unknown_param_is_rejected() -> None:
    with pytest.raises(evalsi.EvaluatorConfigError):
        valid_citations.bind({"min_citation": 2})




def test_records_without_context_are_skipped_not_scored_zero() -> None:
    result = evalsi.evaluate(DATA, [valid_citations])
    by_id = {r.record_id: r for r in result.results}
    assert by_id["no-context"].outcome is evalsi.Outcome.SKIPPED
    summary = result.metric("valid-citations")
    assert (summary.n, summary.skipped, summary.errors) == (3, 1, 0)
    assert summary.mean == pytest.approx(1 / 3)  # only "good" passes

valid_citations.py

Section titled “valid_citations.py”
"""A custom evaluator: does the answer cite its sources correctly?


For a RAG answer such as "Paris is the capital [1]. It lies on the Seine [2].",
the evaluator checks that every ``[n]`` marker points at a retrieved context
passage (1 <= n <= len(context)) and that there are at least ``min_citations``
of them. It is deterministic, needs no judge, and costs nothing to run.
"""


from __future__ import annotations


import re


from evalsi import (
    MetricSpec,
    Pack,
    Record,
    Requirements,
    Score,
    ScoreType,
    SkipRecord,
    evaluator,
)


_MARKER = re.compile(r"\[(\d+)\]")




@evaluator(
    name="acme/valid-citations",
    version="1.0.0",
    description="Every [n] citation in the output points at a retrieved passage.",
    requires=Requirements(output=True, context=True),
    outputs=[MetricSpec("valid-citations", ScoreType.PASSED, higher_is_better=True)],
)
def valid_citations(record: Record, *, min_citations: int = 1) -> Score:
    # Requirements already skip records with no output or no context. Raise
    # SkipRecord for anything else the evaluator cannot judge.
    assert record.output is not None
    text = record.output.as_text()
    if not text.strip():
        raise SkipRecord("empty output")


    cited = [int(n) for n in _MARKER.findall(text)]
    available = len(record.context)
    dangling = sorted({n for n in cited if not 1 <= n <= available})


    if dangling:
        return Score(
            passed=False,
            explanation=f"cites {dangling}, but only {available} passage(s) were retrieved",
        )
    if len(cited) < min_citations:
        return Score(
            passed=False,
            explanation=f"{len(cited)} citation(s), expected at least {min_citations}",
        )
    return Score(passed=True, explanation=f"{len(cited)} valid citation(s)")




# A pack makes the evaluator available by name (`--evaluators acme/valid-citations`,
# servers, online policies) once this package is installed; see pyproject.toml.
PACK = Pack(name="acme", description="Acme checks.", evaluators=[valid_citations])