Evaluator Tool Internal
Production blind-A/B human evaluation web app for the AO Next Benchmark scaffolding ablation. Same 5+5 criteria rubric and 6-tier scoring scale as the paired LLM-as-judge framework, so human and LLM scores are directly comparable.
The Evaluator Tool Internal is a production blind-A/B human evaluation web app that runs the human side of the AO Next Benchmark scaffolding ablation (project page publishing soon). It uses the same 5+5 criteria rubric and 6-tier scoring scale as the paired LLM-as-judge framework, so human and automated scores are directly comparable for cross-method triangulation.
The gap it closes
The AO Next Benchmark LLM-as-judge framework needed a human ground-truth check that did not require routing every comparison through external labeling. External labeling has trade-offs: turnaround latency, lack of internal subject-matter context, and inflexibility when the rubric or dataset changes between studies. The benchmark needed an internal surface where evaluators could grade harness-vs-control pairs against the exact same rubric the LLM judges used, with model identity blinded so the preference data could not be biased by knowing which arm produced which article.
Architecture
Rubric parity with the LLM-as-judge framework
Two scored dimensions, AI Discoverability and Brand Kit Alignment, each with 5 displayed criteria. AI Discoverability: Information Gain, Search Intent Alignment, E-E-A-T, Readability, Citability. Brand Kit Alignment: Tone and Voice, Writing Rules, Audience Fit, Style Guide Adherence, Overall Brand Adherence. Both dimensions scored on the same 6-tier 0.0 to 1.0 scale (0.0, 0.2, 0.4, 0.6, 0.8, 1.0) used by the LLM-as-judge framework. Aggregate scores are computed server-side on submission, not entered by the evaluator.
Blind A/B with hidden key
Evaluators see only Article A and Article B. The hidden key mapping each pair’s articles back to model-a / model-b (treatment vs. control) is stored alongside each submission for downstream analysis but never displayed to the evaluator. Preference (A or B) and per-dimension scores are submitted together.
Plugin-style dataset registration
Adding a new brand dataset is a four-step recipe: drop a condensed brand kit, drop the pair folder with its key, register the folder in the datasets list, and run the brand-kit injection script. The criteria panel auto-resolves the correct brand kit per pair, so evaluators always grade against the right brand context without manual configuration.
Smart prioritization
Ungraded pairs surface first, reshuffled daily so the same pair does not always appear at the top of every evaluator’s queue. Once all pairs in a dataset have at least one grade, graded pairs are surfaced again for additional ratings. After each submission the next ungraded pair auto-loads, removing manual navigation friction.
What shipped
160 article pairs evaluated across two brand datasets that varied in whether the harness tool and context layer was enabled during generation. The human preference data converged with the LLM-as-judge result: enabling the harness layer’s tool and context affordances increases the human preference rate for the harness arm; stripping those affordances reverses it. The durable internal surface for human-checking future scaffolding and model comparisons.