Robust Comparison And Ranking

Robust comparison is the public-descent audit for leaderboards and rankings. Instead of auditing a single scalar target, it checks pairwise margins among alternatives in long-form data under the same admissible recomposition.

Why Pairwise Margins Matter

Do not certify a ranking by comparing independent item intervals. Independent intervals allow each alternative to move under a different hidden mix, which is usually not the comparison being reported.

For alternatives A and B, the robust comparison target is the cellwise margin:

score(A, hidden cell) - score(B, hidden cell)

For “lower is better” metrics such as loss, the sign is reversed. The pairwise ordering is certified when the lower endpoint of that margin interval stays above the chosen margin threshold, usually zero.

Minimal Example

import updatesupport as us

report = us.robust_comparison_report(
    rows,
    item="model",
    public=["task_family", "difficulty"],
    hidden=["task_family", "difficulty", "source", "prompt_template"],
    target="score",
    weight="n",
    q="saturated",
    higher_is_better=True,
)

print(report.to_markdown())

The input should be long-form: one row per alternative and hidden cell, or raw rows that aggregate to that shape.

Report Status

The report status has three levels:

• full_ranking_stable: every observed pairwise ordering is certified,

• winner_stable: the observed winner is certified, but some lower-ranked pairwise ordering is not,

• ambiguous_winner: at least one challenger can become competitive with the observed winner under the selected stress test.

Lower-Is-Better Metrics

For losses, error rates, latency, default rates, or other metrics where smaller values are better:

report = us.robust_ranking_report(
    rows,
    item="model",
    public=["segment"],
    hidden=["segment", "scenario"],
    target="loss",
    weight="n",
    higher_is_better=False,
)

robust_ranking_report() is an alias for robust_comparison_report().

Structured Exports

Robust comparison reports support the same artifact paths as other reports:

payload = report.to_json()
tables = report.to_tables()
frames = report.to_dataframes()

The main tables are:

• summary: winner, certified winner, status, Q, and representation metadata,

• items: observed value and hidden-composition interval per alternative,

• pairwise_margins: margin interval and certification status for each observed pairwise ordering.

Current Scope

The first API slice assumes a balanced comparison design: every alternative has the same retained hidden-cell support and the same hidden-cell weights. That is the clean setting for model leaderboards, benchmark slices, A/B variants scored on the same segment cells, and other common comparison reports.