Skip to content
Evals.siEvals.si

0004. Self-hosted in the client's environment; hosted multi-tenant later

  • Status: Accepted, 2026-10-05 (design plan D5)

Decision

Section titled “Decision”

For now, Evals.si is installed and operated by clients inside their own infrastructure. A hosted multi-tenant offering comes later, when there is compute for it.

Consequences

Section titled “Consequences”
  • Nothing calls home. Telemetry is off unless the operator turns it on.
  • Bring your own everything:
    • models and judges, with no default judge;
    • storage (Postgres, ClickHouse, S3-compatible);
    • identity (OIDC);
    • secrets;
    • observability.
  • Air-gapped installs are supported. Phase 4 shipped the offline bundle (deploy/airgap), tested in CI by installing from it with no registry access. The dataset mirror tool is not built yet.
  • Least-privilege install. The main Helm chart (evalsi) is namespace-scoped, and cluster-scoped pieces live in separate charts (evalsi-crds, evalsi-sandboxd).
  • Single tenant per install, with projects inside it. Every project-scoped row carries project_id and a reserved tenant_id column (empty for now), so a hosted multi-tenant service can fill it in rather than reshape stored data.