Elegant

Evals are tests for the things unit tests can't check

The Elegant team 5 min readai-and-frontend
Evals are tests for the things unit tests can't check

A unit test checks a computable fact: given this input, does the function return this exact value? Enormous amounts of software quality are not computable that way. Is this explanation clear? Is this component accessible in spirit, not just in attributes? Did the AI’s answer actually solve the problem, or just look like it did? An eval is a test for those — a repeatable check against a standard you can describe but not reduce to ===. As AI writes more of our prose, designs, and code, evals become as important as unit tests, because the questions worth asking about AI output are almost all of the un-computable kind.

Unit tests check computable equality; evals check describable standards Left: a unit test comparing output to an exact expected value. Right: an eval judging an output against a rubric or a stronger judge, producing a graded verdict. unit test output === expected ? computable, exact, pass/fail eval output vs rubric / judge describable standard, graded verdict runs on every change, like a test
A unit test asks "does it equal?"; an eval asks "does it meet the standard?" Both run repeatably on every change — the eval just judges instead of comparing.

When the answer isn’t computable, write a rubric

The core move is to make the un-computable checkable by writing down the standard as a rubric — the criteria a human would use — and then applying it repeatably. For an AI-authored explanation, “is this genuine or slop?” becomes a set of concrete tests:

// eval: encode the standard you can describe, then check it on every draft
function evalExplanation(text) {
  return {
    hasConcreteExample: /```/.test(text),                 // shows, not just tells
    takesAPosition: !/it depends|there are many ways/i.test(text),  // commits
    rightLength: wordCount(text) >= 400,
    passed() { return this.hasConcreteExample && this.takesAPosition && this.rightLength; },
  };
}

Some criteria are mechanical like these; others need a judge — a stronger model or a human — scoring against the rubric. Either way, the standard is written down and applied the same way every time, which is what makes it a test and not a vibe.

The eval must be able to fail the thing it checks

An eval is only meaningful if it can say no. A slop-detector that passes everything is not a check; it is decoration. So a good eval is validated in both directions: give it a known-bad output and confirm it fails, give it a known-good one and confirm it passes:

evalExplanation("It depends. There are many approaches.").passed();  // false — good, it rejects slop
evalExplanation(realPostWithCodeAndPosition).passed();               // true  — good, it accepts quality

An eval you never watched reject something is an eval you cannot trust to reject the next thing.

Evals are the harness for the un-computable

The reason this matters now is throughput and subject matter: AI produces a flood of outputs whose quality is exactly the describable-but-not-computable kind — is the prose good, is the design clear, did the answer really work. Human review does not scale to that flood, and unit tests cannot express the question. Evals fill the gap: a repeatable, describable standard that runs on every output like a test suite runs on every commit, folded into the composite alongside the mechanical checks (one signal, not the only one, since a judge can be wrong). This is precisely how a content guardrail keeps AI-written material honest — the slop verdict fails the draft, so the next one improves because the writing did, not because the check was loosened. The harness-skill-eval exercise builds exactly this: the eval that says what “correct” means for an output no === can grade.