Generation Is Not Grading

Who decides whether a model understood a sentence correctly? Not the model that produced the understanding.

Wellzai's evaluation harness for intent understanding states this as a rule. A model may help create synthetic test sentences, but the same model must not become the sole judge of whether its own interpretation is correct. Generation is not grading. Behind the rule sits an explicit order of evaluation authority: structural assertions first, then curated gold semantics, then deterministic semantic invariants, then human-reviewed equivalent interpretations, and a model judge last, only for genuinely fuzzy comparisons, optional and subordinate. Cases that cannot be decided reliably go to a review queue instead of being silently scored either way.

Two further rules protect the grading itself. Expected answers are treated like specifications: a failing test must not be "fixed" by changing the expected semantics without explicit review. And every confirmed interpretation failure is to become a permanent regression case before the logic is fixed, so the failure is preserved even after it stops happening. The harness keeps its original example, a constructed semantic test oracle about 600 acres and financing, in that corpus for good, because it exposed a misreading in Wellzai's intent understanding.

The implication is that an evaluation suite is not only a test of the system. It is a record of what has been decided "correct" means, and it deserves the same governance as the system it tests.

The open question is the old one, sharpened. Curated gold is written by people; deterministic invariants encode people's decisions; reviewers can disagree. The hierarchy pushes authority toward what is explicit and reviewable, but it does not make the top of it infallible. Who grades the graders, and how would anyone notice when the gold itself is wrong?