Portal
Documentation: all sections

Evals

Evals turn a model into something you can regression-test. You define a suite of cases, each with assertions about the output, run the suite against a model, and get a pass rate back. Run it again later and Saxeo flags a regression when the pass rate dropped versus the previous run.

Runs are real inference: each case is a genuine metered call on your prepaid balance, so the pass rate reflects the model as it actually behaves. What is stored is only the score, the pass/fail of each case, and which assertions failed. The model output itself is never stored, in keeping with the privacy contract.

Evals are session-authed: they run behind your Sign-In With Ethereum dashboard session, and there is a portal page for them at Portal → Evals.

Assertions

Each case carries one or more assertions. An assertion has a type and a value:

containsPasses when the output...contains value as a substring (case-sensitive).
icontainsPasses when the output...contains value, case-insensitively.
not_containsPasses when the output...does not contain value.
equalsPasses when the output...equals value exactly.
regexPasses when the output...matches the regular expression value.
min_lengthPasses when the output...is at least value characters long.

A case passes only when all of its assertions pass.

Quickstart

# 1. Create a suite
curl https://www.saxeonetwork.tech/__api/v1/evals \
-H "authorization: Bearer $SAXEO_SESSION_TOKEN" \
-H 'content-type: application/json' \
-d '{
  "name": "support-tone",
  "model": "saxeo",
  "cases": [
    {
      "input": "A customer asks for a refund. Reply in one sentence.",
      "assertions": [
        {"type": "icontains", "value": "refund"},
        {"type": "min_length", "value": 20}
      ]
    }
  ]
}'

# 2. Run it (real, metered inference)
curl -X POST https://www.saxeonetwork.tech/__api/v1/evals/$EVAL_ID/run \
-H "authorization: Bearer $SAXEO_SESSION_TOKEN"

A run reports the pass rate, whether it regressed, the cost, and the per-case outcome, including which assertions failed:

{
  "run_id": "run_5f2a…",
  "total": 1,
  "passed": 1,
  "pass_rate": 1.0,
  "regression": false,
  "cost_micro_usd": 240,
  "results": [
    { "index": 0, "passed": true, "failed_assertions": [] }
  ]
}

Tracking regressions over time

GET /v1/evals/{id}/runs returns the run history with each run's pass rate, so you can watch a suite over time and see exactly when a model change moved the number. The regression flag on a run is set whenever its pass rate is lower than the run before it, which is the signal to look at what changed.

Endpoints

All session-authed (Authorization: Bearer sess_…).

POSTPath/v1/evalsWhat it doesCreate a suite. Body {name, model, cases}.
GETPath/v1/evalsWhat it doesList your suites.
DELETEPath/v1/evals/{id}What it doesDelete a suite.
POSTPath/v1/evals/{id}/runWhat it doesRun the suite against its model. Returns pass rate, regression, cost, and per-case results.
GETPath/v1/evals/{id}/runsWhat it doesRun history with pass rate over time.

Privacy and limits