Agent evaluation benchmark [supplemental]
Control-arm execution platform for the AO Next Benchmark scaffolding ablation. Resolves grid inputs into Playbook input variables, renders staged prompts that preserve Playbook stage bodies verbatim, runs them automatically as a single-chat baseline with per-stage instrumentation, and generates harness-vs-control article pair files for downstream human preference labeling.
![Agent evaluation benchmark [supplemental]](/assets/portfolio/depends-agent-benchmark.png)
Agent benchmark is the control-arm execution platform for the AO Next Benchmark scaffolding ablation. It resolves grid inputs into Playbook input variables, executes the control arm either manually as staged prompts or automatically as a single-chat baseline, captures per-stage telemetry, and generates the harness-vs-control article pair files that downstream human preference labeling consumes.
The gap it closes
The AO Next Benchmark scaffolding ablation only works if the control arm mirrors the harness on every variable that is supposed to be constant. Same Playbook stage instructions, same resolved input variables, same article scope. Only the execution environment can vary. Generating the control arm by hand at scale breaks the experiment in non-obvious ways: paraphrased stage bodies, inconsistent input mapping between rows, missing file-backed inputs, no provenance trail back to the source row. The control arm needs a deterministic execution tool, not a process.
Architecture
An orchestrator package with three jobs.
Input variable resolution
Loads one row from the benchmark grid by index or row id, maps the evaluated inputs onto the Playbook’s declared input names, injects file-backed fields (such as the brand kit document) via path references, and returns the resolved inputs. A separate module parses the Playbook markdown into structured stages, inputs, and artifacts. A hardcoded Playbook is held in-process so execution does not depend on an external markdown file at runtime, with an override flag available for development.
Manual execution path
Writes a per-run package for manual execution in a conversational LLM chat platform: a context prompt with resolved inputs, one prompt per Playbook stage with the stage body preserved verbatim (minus tool-access lines, since the control arm has no tool access), a manual runbook, artifact placeholders, and a run manifest.
Automated execution path
Runs the Playbook as a single continuous chat against the chosen generator model. The system prompt is only the Playbook frontmatter description. Each Playbook stage is a separate user turn. Prior stage outputs live as assistant turns in the conversation, not re-pasted. Each stage’s user turn includes only the inputs that the stage body references (just-in-time context). No tools, no subagents, no knowledge base. Faithful to a careful manual conversational-LLM run.
Per-stage token usage and timing are captured per run. The full conversation is captured for debugging. Prompt caching is marked on the latest user turn. Concurrency runs through a bounded worker pool.
Output and handoff to human labeling
Initializes a per-run output CSV and updates the artifact cell after each stage completes. Pairs the harness article with the control-arm article on each row, then emits one pair file per row in blind A/B labeling format with model-a / model-b assignment randomized.
Reproducibility
Deterministic run folders keyed by row id. Per-run manifests link each run to its source row, Playbook name, execution mode, and resolved input snapshot. Any benchmark row can be traced back to its source and reconstructed from manifest data.
The pipeline is model-agnostic by design. The single-chat architecture (sequential turns, just-in-time inputs, no tool access) generalizes to any conversational LLM that supports system / user / assistant turns. Generator models swap without changing the orchestration.
What shipped
Deterministic control-arm generation for the AO Next Benchmark across multiple datasets. Both manual and automated execution paths shipped. Source-definition discipline encoded in per-run manifests so any benchmark row can be traced back to its source row, Playbook name, execution mode, and resolved inputs.