Experimental · v2.0.0

Agent Eval Bench

Define up to 8 test cases (input + expected traits). Paste two system prompt variants. Score them 1–5 on instruction-following, brevity, citations, and refusal quality, with notes. Suites persist in localStorage. Phase 1 uses a single model call when an API key exists; otherwise it mock-scores with an honest disclaimer — never a fake ‘GPT’ result dressed as real.

diff.ghost · 1,762 votes · 10,694 score · 5,408 opens

#evals#prompts#agents#rubric

Spec

Build an Agent Eval Bench — a tiny harness, not a MLOps platform.

PRODUCT
The user defines up to 8 cases (input + expected traits). They paste two system prompts (A / B). The bench scores each case on a rubric and shows a side-by-side. Honesty about how scoring happened is part of the product.

RUBRIC (fixed, 1–5)
- Instruction-following
- Brevity
- Citations (or ‘n/a’ if the case does not ask)
- Refusal quality (or ‘n/a’)
Notes: 1–2 sentences per cell, pointing at the output.

SCORING BACKEND (Phase 1)
- If a server-side model is available, use one user-initiated run (a “Run eval” button, never on page load). Cap cases at 8, cap output tokens reasonably.
- If not available, mock-score with a clearly labeled “HEURISTIC — not a model” banner. Heuristic may look at length, whether expected keywords appear, whether a refusal was asked. Never pretend to be a named lab’s model.

SCREENS
1. Suite — name, cases editor (add/remove, 1–8)
2. Variants — prompt A, prompt B
3. Run — matrix: cases × (A|B) × rubric. Totals. Winner line only if the delta is >0.4 average; otherwise “too close to call”.
4. Case drill — full outputs.

DATA
Suites in localStorage. Last run results too.

VISUAL
Lab bench. Matrix in mono. A/B as two columns, not two colors of the rainbow. Banner for heuristic mode is impossible to miss.

OUT OF SCOPE
CI integration, dataset marketplace, multi-user orgs, leaderboard of models.

ACCEPTANCE
- Cannot add a 9th case.
- Heuristic mode banner is visible when no model run happened.
- Totals match the visible cell scores.
- Mobile: matrix scrolls horizontally inside a labeled scroller (the one allowed overflow), with a hint “swipe the grid”.