Methodology

This page documents the conventions used by the SOTA Dashboard: how the Composite Agent Score is computed, what counts as a Frontier / Agent-Product / Edge-SLM model, how the Pareto frontier is drawn, and what each of the 14 Agent widgets measures. If you only have five minutes, the “Composite Agent Score” section below covers 80% of what you need.

Composite Agent Score

The Composite Agent Score collapses a model’s performance across all tracked agentic benchmarks into a single 0–100 number. It is computed client-side in dashboard/js/agent-charts.js by the _composite() function (mirrored in Agent._compositeScores() in agent.js).

Formula

For a model m with scores on a set of benchmarks B(m):

For each benchmark b in B(m):
    max_b   = max value reported for b across all models
    v       = m's value on b
    if b is in CATEGORIES[*].lower_better:
        norm_b = (1 - v / max_b) * 100      # invert "lower is better"
    else:
        norm_b = (v / max_b) * 100           # higher-is-better

agent_score(m) = mean( norm_b  for all b in B(m) )
coverage(m)    = |B(m)|

In words: each benchmark is rescaled to 0–100 by dividing by the highest value any model has achieved on that benchmark, then we take the simple mean of all per-benchmark scores. Lower-is-better metrics (jailbreak ASR, scheming subversion rate, etc.) are inverted first so “higher = better” on the final scale.

Coverage threshold

A model is shown on rankings, scatter plots, and leaderboards only when it has scores on ≥ 3 distinct agentic benchmarks. Models with 1–2 benchmarks are excluded to avoid noisy outliers from cherry-picked single-benchmark wins. The threshold is enforced in _composite() at the line if (byModel[midKey].count < 3) continue;.

Lower-better benchmarks (inverted before averaging)

The following benchmark IDs are listed in CATEGORIES.safety.lower_better and inverted before they contribute to the composite:

• agentdojo_targeted_asr — AgentDojo prompt-injection attack success rate
• injecagent — InjecAgent indirect injection success rate
• browserart — BrowserART browser-agent jailbreak rate
• anthropic_shade_browser_asr_attempts
• anthropic_shade_browser_asr_scenarios
• anthropic_shade_browser_asr_with_safeguards
• agent_red_teaming — red-team success rate
• agentsmith_inf
• apollo_scheming_oversight_subversion
• apollo_scheming_persistence

Model Class Definitions

Every model in the dashboard is tagged with one of three classes. The Agent tab’s W4 Frontier-Product-Edge dot plot uses these classes directly; the W1 cost scatter colors bubbles by class.

Strict Score Attribution Policy

Every score in data/scores.json must satisfy this rule:

The following are explicitly not admissible and have been purged from the dataset:

• “Frontier baseline” placeholders that approximate a model’s likely performance.
• Linear or log extrapolations from older versions of the same model family.
• Numbers cited “from a web report” without a working primary URL.
• Numbers attached to the wrong model (e.g. a Claude Opus 4 score reported on a Claude Sonnet 4.5 row).
• Rounded headline numbers from press releases that conflict with the model card’s exact value.

Each row in scores.json carries a source_url field. The Score History modal opens that URL when a user clicks a score, so the chain of evidence is always one click away.

Pareto Frontier Definition

The W1 Cost vs Performance Scatter draws a dashed line through the Pareto-efficient set: the subset of models for which no other model is strictly better on both axes simultaneously.

Formally, given points p_i = (cost_i, score_i):

p_i is on the frontier  ⇔
    no p_j  exists with
        cost_j ≤ cost_i  AND  score_j ≥ score_i
        AND  (cost_j < cost_i  OR  score_j > score_i)

In plain language: a point is on the frontier if you cannot find another point that is at least as cheap AND at least as good, with at least one of those being strictly true. The dashed line connects frontier points sorted left-to-right by ascending cost. Implementation lives in _paretoFrontier() in agent-charts.js.

Edge SLM Cost Treatment

• W1 Cost vs Performance Scatter: Edge SLMs run free on-device, so they have no per-token API price. They are plotted at the placeholder X = $0.01 / 1M output tokens (a log-scale-friendly value defined as EDGE_X_PLACEHOLDER). A note in the chart legend tells readers this is a visualization convention, not a real price.
• W12 Cost Simulator: Edge SLMs are counted as $0/month regardless of token volume. Compute, electricity, and device amortization are out of scope for the simulator — the goal is API-cost comparison, and on-device inference simply does not have an API bill.

14 Widget Inventory

The Agent tab is built from 14 named widgets. Each widget has a single canonical purpose:

Versioning & Updates

• Score updates are recorded in data/export/reports/changelog.json. Each entry tags the model, benchmark, old → new value, and source URL.
• Session-level work (waves, sweeps, audits) is summarized in HISTORY.md at the repository root.
• Ingest rules (file naming, source priority, duplicate handling) are documented in CONTRIBUTING.md.
• Cache busting: each JS <script> tag carries a ?v=YYYYMMDD<letter> query string. When the deploy workflow ships a new build, it stamps the version with git rev-parse HEAD so browsers always see the latest code.

Source code references

• dashboard/js/agent-charts.js — _composite(), _paretoFrontier(), all 14 widgets.
• dashboard/js/agent.js — CATEGORIES, AGENT_PRODUCTS, EDGE_SLMS, _allAgentBenchmarks().
• data/scores.json — one row per (model, benchmark, value, source_url) tuple.
• data/aa_pricing.json — per-model input/output token prices used by W1 and W12.
• data/export/reports/changelog.json — score-update audit log.