Decision records
Short records of decisions that shape the code. Each one says what was decided, why, and what follows from it. The design plan (../DESIGN.md, §22) lists the decisions that are still open.
| # | Decision | Status |
|---|---|---|
| 0001 | First users: agent builders, agent platform builders, LLM app developers | Accepted |
| 0002 | Go core, Python runtime | Accepted |
| 0003 | No web UI for now | Accepted; revisited by 0014 (a read-only UI) |
| 0004 | Self-hosted in the client’s environment; hosted multi-tenant later | Accepted |
| 0005 | Sandbox isolation ladder: Firecracker, then bubblewrap or Landlock, then a hardened pod | Accepted, implemented |
| 0006 | Names: evalsi, evalsid, evals.si, evalsi.v1alpha1 |
Accepted |
| 0007 | Testing the sandbox rungs | Accepted |
| 0008 | Agent platform builders consume Evals.si as a service | Accepted |
| 0009 | Standalone: SQLite and an in-process scheduler; NATS with Kubernetes, DuckDB later | Accepted, implemented (DuckDB deferred) |
| 0010 | Identity and access (OIDC/JWT, API keys, RBAC with custom roles, CEL rules) come next, modeled on agentgateway | Accepted, implemented |
| 0011 | Agent environments and benchmark grading: upstream scoring, direct format import, Firecracker without a network device | Accepted, implemented |
| 0012 | Kubernetes: the operator as an API client, one image, three charts, service-account tokens as OIDC | Accepted, implemented |
| 0013 | Rewards from evaluators, composed in Python and Go with shared vectors; checkpoints served outside the operator | Accepted, implemented |
| 0014 | MCP locally and on evalsid; guardrails on agentgateway’s protocols; Wasm plugins on wazero; a web UI (read-only, then annotating and browser sign-in) | Accepted, implemented |
| 0015 | Worker secrets named in specs are granted per project and destination host (HTTPS), judges scoped per project; refusals audited, admission checks, reload | Accepted, implemented |
| 0016 | Trace-source connectors: a TraceSource per store, a Go Connector interface, per-variant MLflow transports, start-time watermarks with lookback, a durable IngestBatchContext, write-back as assessments, egress by chart value |
Accepted, partly built (Langfuse 4.56.0 live-verified; Databricks experiment transport contract-tested only) |
| 0017 | Charts name what they create after the release; the default release keeps its names; dependencies bundled or external | Accepted, implemented |
| 0018 | The isolated headless dsh small-repo demo explicitly relies on Evals.si’s outer sandbox; standalone defaults remain confined | Accepted, implemented |
| 0019 | Server-owned trace redaction before push/pull persistence and evaluation, including source overrides | Accepted, implemented |
| 0020 | Durable keyed retries without secret replay, bounded dataset publication, terminal deletion and scheduled batch windows | Accepted, implemented |
| 0021 | Sandbox time is a daily per-project quota (sandbox_seconds_per_day), metered in the run manager from agent task and sandbox-pool evaluator wall time |
Accepted, implemented |
| 0022 | Cross-run slices are computed on the server (SliceRuns), in Go over each run’s stored results; arbitrary SQL stays on the client |
Accepted, implemented |
| 0023 | Archiving keeps a run’s summary and drops its records, outputs and results; runs.archive_after does it automatically, and automatic deletes and archives are audited |
Accepted, implemented |
| 0024 | evalsi ships Linux platform wheels that bundle evalsid and bwrap; EVALSID, then the bundle, then PATH |
Accepted, implemented |
| 0025 | A CheckpointServer resource has the operator run vLLM for a checkpoint; the watcher creates and deletes one per checkpoint through kubectl |
Accepted, implemented; not yet run on a GPU cluster |
To add one, copy the shape of an existing record, take the next number, and link it here.