A product team runs an A/B evaluation between two LLM configurations — say, GPT-4.1 versus a fine-tuned Llama variant — on a 500-question internal benchmark. Model B scores 2.3 percentage points higher. A quick binomial test returns p = 0.03. The team declares victory, ships the new model, and moves on.

Six weeks later, user complaints roll in. The "better" model is actually worse on the long tail of real traffic — multi-turn conversations, ambiguous instructions, edge-case formatting. What happened?

This is one of the most common failure modes in LLM evaluation today, and it has almost nothing to do with the math behind the p-value and everything to do with what the p-value was never designed to tell you. A p-value answers "how surprising would this difference be if there were truly no difference at all?" It does not answer "how large is the difference, and does that size matter for my product?" Those are different questions, and conflating them is now a structural problem across AI benchmarking — not a rookie mistake, but a systemic one baked into how leaderboards, release notes, and vendor comparisons are built and reported.

This piece is about the second question: effect size. We'll walk through why p-values mislead in LLM evaluation specifically, what effect-size metrics are actually usable for benchmark data, what the public research and leaderboard literature suggests about typical effect magnitudes, and how a PM or analyst should structure a benchmark report so that a reader — including a skeptical one — can tell the difference between "statistically detectable" and "practically meaningful."

Why P-Values Break Down Specifically in LLM Benchmarking

Null-hypothesis significance testing was built for a world of large, independent, i.i.d. samples and binary pass/fail outcomes — clinical trials, agricultural experiments, manufacturing defect rates. LLM evaluation violates almost every one of those assumptions, and it does so in ways that systematically push p-values toward "significant" even when the underlying effect is small or illusory.

1. Benchmark sizes are tiny relative to clinical-trial norms, but that cuts both ways. Many widely cited benchmarks — GPQA Diamond (198 questions), parts of MMLU-Pro, HumanEval (164 problems) — have sample sizes an order of magnitude smaller than what a biostatistician would accept for a comparable claim. Small n inflates the variance of the estimated accuracy difference, which should make significance harder to reach, not easier. The danger is the opposite direction: teams see a non-significant result on a small benchmark and interpret "not significant" as "no difference," when in fact the confidence interval is simply too wide to detect a difference of real-world importance. This is the classic absence-of-evidence-vs-evidence-of-absence trap, and it's endemic in benchmark write-ups that report accuracy deltas without intervals.

2. Items are not independent, and model errors are correlated across questions. Standard binomial or chi-squared tests on aggregate accuracy assume each question is an independent Bernoulli trial. In practice, a model that struggles with multi-step arithmetic will miss many related questions in a benchmark that over-samples a particular skill, inflating apparent sample size while actually providing much less independent information than the raw item count suggests. This violates the independence assumption underlying standard error calculations and tends to make p-values too small — overstating confidence. The paired nature of most model-vs-model comparisons (same questions, two models) argues for McNemar's test or a paired bootstrap rather than two independent-sample tests, yet a lot of public benchmark reporting still uses the latter.

3. Benchmark contamination and prompt sensitivity add a variance source no significance test captures. A model's score on a fixed benchmark depends heavily on prompt template, few-shot formatting, and decoding parameters (temperature, sampling vs. greedy). Published work on prompt sensitivity — including analyses from EleutherAI's lm-evaluation-harness maintainers and independent reproducibility studies — has documented accuracy swings of several points on the same model from formatting choices alone. See the lm-evaluation-harness documentation and associated discussion: https://github.com/EleutherAI/lm-evaluation-harness. A p-value computed on one fixed run says nothing about this source of variance, because it only ever measures sampling noise within one evaluation configuration.

4. LLM-as-judge scoring introduces evaluator-level variance that significance tests on a single judge call ignore. When benchmark labels or pairwise preferences are produced by another LLM rather than a fixed ground truth (as in Chatbot Arena-style pairwise setups or MT-Bench), the "measurement" itself has a noise distribution — judge model, judge prompt, and even judge temperature all shift results. The Chatbot Arena team's own methodology writing acknowledges this and uses Bradley-Terry models with bootstrapped confidence intervals specifically because a naive win-rate percentage with a p-value would overstate certainty. See the LMSYS Chatbot Arena methodology notes: https://lmsys.org/blog/2023-05-03-arena/ and the associated paper, "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference," https://arxiv.org/abs/2403.04132.

5. Multiple comparisons are the norm, not the exception, and are rarely corrected. A typical model release touches a dozen or more benchmarks (MMLU, GSM8K, HumanEval, GPQA, MATH, IFEval, BBH, and so on). Running significance tests on each without correcting for multiple comparisons guarantees some benchmarks will cross the p < 0.05 threshold by chance alone. Vendor release blogs that cherry-pick the subset of benchmarks where their model "wins significantly" are implicitly running this exact uncorrected multiple-comparisons procedure, whether or not they frame it in classical-statistics language.

Put together, these five issues mean a p-value on an LLM benchmark comparison is frequently either too optimistic (correlated items, judge noise papered over) or too conservative in a way teams misread as "no difference" (small n, wide intervals ignored). Neither failure mode is fixed by computing the p-value more carefully. Both are fixed by reporting effect size and uncertainty directly.

What Effect Size Actually Means for Benchmark Data

"Effect size" is a family of metrics, not one number, and picking the wrong one for discrete accuracy-style data is its own source of confusion. Four are relevant to LLM evaluation:

Raw accuracy difference (Δ accuracy). The simplest and most interpretable: Model B scored 71.2%, Model A scored 68.9%, Δ = 2.3 points. This is almost always worth reporting as the headline number because it maps directly to "how many more users get a correct answer," but it needs a confidence interval attached — a bootstrap or Wilson-score interval over the paired differences, not just the point estimate.

Cohen's h (for proportions). When accuracy is a proportion, Cohen's h = 2·arcsin(√p₁) − 2·arcsin(√p₂) is the standard effect-size transform because raw percentage-point differences compress or expand depending on where they sit on the 0–1 scale (a 2-point gap near 50% accuracy is not equivalent in information terms to a 2-point gap near 95% accuracy). Cohen's original conventions treat h ≈ 0.2 as small, 0.5 as medium, 0.8 as large — conventions from Jacob Cohen's "Statistical Power Analysis for the Behavioral Sciences" (1988), still the standard reference: https://www.utstat.toronto.edu/~brunner/oldclass/378f16/readings/CohenPower.pdf.

Cohen's d / standardized mean difference (for continuous scores). For benchmarks scored on a continuous scale — judge-assigned Likert scores, embedding-similarity metrics, graded rubric scores out of 10 — Cohen's d expresses the gap in units of pooled standard deviation. This is the metric most directly imported from psychology and medicine, and it's the one most likely to be miscalibrated for LLM work if applied naively, because LLM output distributions are frequently non-normal, multimodal, and heavy-tailed in ways that violate d's underlying assumptions.

Win rate / Bradley-Terry strength (for pairwise preference data). Chatbot Arena-style evaluation doesn't produce a per-item accuracy at all; it produces pairwise "A beats B" judgments aggregated into a latent strength parameter. The natural effect-size unit here is the Elo-like rating gap or, more rigorously, the Bradley-Terry log-odds difference, which can be converted into an implied win probability. A 50-point Arena Elo gap implies roughly a 57% win probability for the higher-rated model in a head-to-head — a number worth sitting with, because it is a far less dramatic edge than leaderboard rank position (#3 vs. #7) visually suggests.

The through-line across all four: every one of them answers "how big," and every one needs a companion confidence interval or credible interval to answer "how sure." A p-value collapses both questions into a single threshold decision. Effect size plus interval keeps them separate, which is exactly what a benchmark consumer needs to make a deployment decision.

What the Public Evidence Actually Shows About Effect Magnitudes

It's worth being honest about the state of the evidence here: rigorous, pre-registered effect-size reporting is still rare in LLM benchmarking, and most numbers below should be read as reported ranges from available literature and leaderboard documentation, not precise point estimates you can cite with full confidence. Several patterns are nonetheless well enough documented to be useful.

Benchmark variance from decoding and prompt-formatting choices alone has been reported in the reported range of 1–5 accuracy points on multiple-choice tasks, based on reproducibility analyses circulated around the lm-evaluation-harness project and discussions of benchmark instability by groups including Hugging Face's Open LLM Leaderboard maintainers, who documented enough non-reproducibility to prompt a leaderboard redesign. See Hugging Face's own retrospective: https://huggingface.co/spaces/open-llm-leaderboard/blog and related community discussion on benchmark reproducibility. If your model comparison shows a 2-point gap, and prompt formatting alone can move a score 1–5 points, the comparison is not telling you what you think it's telling you unless both models were evaluated under identical, controlled harness settings.

Chatbot Arena's own public methodology discussion repeatedly notes that many adjacent leaderboard positions are not statistically distinguishable at typical vote volumes — a point the LMSYS team has made explicitly in blog commentary accompanying rating updates, where confidence intervals on Arena scores for closely-ranked models visibly overlap: https://lmsys.org/blog/2023-05-03-arena/. The practical implication: a leaderboard rank difference of two or three positions among closely clustered frontier models is frequently within noise, even though the leaderboard UI presents a strict ordering.

On the "LLM-as-judge" reliability question, the paper introducing MT-Bench and the LLM-judge paradigm — "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (Zheng et al., 2023) — reports agreement rates between GPT-4-as-judge and human preference in the reported range of roughly 80–85% on their test set, which is high but meaningfully below 100%, implying a non-trivial misclassification rate baked into any benchmark that uses LLM judges as ground truth: https://arxiv.org/abs/2306.05685. Any "Model A beats Model B" claim resting on judge-based scoring inherits this ceiling on reliability, and the honest move is to propagate that uncertainty into the reported effect size rather than treating judge output as error-free ground truth.

For task-specific benchmarks like GSM8K and MATH, published model comparison tables across major model cards (OpenAI, Anthropic, Google, Meta model cards for GPT-4-class, Claude, Gemini, and Llama releases) frequently show close competitors within a reported range of 1–3 points on these benchmarks at the frontier — differences that are plausible but rarely accompanied by any confidence interval in the original release materials, meaning readers cannot tell from the official numbers alone whether the gap is reproducible across prompt variants or sampling seeds.

The consistent theme across all of these sources: typical "headline" differences touted in release comparisons (low single-digit percentage points) are frequently on the same order of magnitude as the measurement noise introduced by prompt formatting, judge variance, and sampling — which is exactly why effect-size-with-interval reporting matters so much more here than in domains where measurement noise is small relative to effect size.

Effect Snapshot: Illustrative Comparisons

The table below is a structured illustration of how effect-size reporting should look in practice. Values marked "reported range" draw on the sources discussed above; they are not precise figures to cite as exact statistics, and in a real evaluation report every cell here should be replaced with your own computed intervals, not borrowed numbers.

Intervention / Comparison Metric Effect Size Confidence Context / Caveat
Frontier model A vs. B on MMLU-style multiple choice Accuracy delta Reported range: 1–3 pts Low–Medium — rarely paired with CIs in model cards Gap is comparable in magnitude to prompt-format-induced variance documented by lm-eval-harness community
Prompt template / few-shot formatting change (same model) Accuracy delta Reported range: 1–5 pts Medium — documented across multiple reproducibility writeups This alone can exceed many vendor-reported "wins"; always hold formatting constant across compared models
Fine-tuning on domain-specific instruction data Task accuracy (domain-specific eval) Reported range: 5–15 pts Medium-High when paired bootstrap used Large effects are real for narrow, well-targeted fine-tunes but rarely transfer to general capability
Chain-of-thought prompting vs. direct prompting Accuracy on reasoning benchmarks (e.g., GSM8K-style) Reported range: 10–30 pts, highly task-dependent Medium — effect well-replicated directionally, magnitude varies by task One of the more robust, large, and reproducible effects in the prompting literature; see Wei et al. 2022 chain-of-thought paper
LLM-judge vs. human preference agreement Agreement rate Reported range: ~80–85% Medium — single benchmark (MT-Bench), not universally replicated across domains Implies ~15–20% judge "error rate" that should be treated as measurement noise on any judge-scored benchmark
Adjacent Chatbot Arena leaderboard rank (2–3 positions apart) Win probability implied by Elo gap Often ~52–58% implied win rate Low — LMSYS itself flags overlapping intervals for close ranks A "higher ranked" model is not reliably better in practice at this margin; treat as a tie
Context window extension / RAG retrieval improvement Task-specific accuracy on long-context QA Reported range: 3–20 pts, task-dependent Low-Medium — highly sensitive to eval design and retrieval quality Magnitude swings enormously with benchmark construction; treat vendor-reported top-end numbers skeptically

Sources for directional patterns: lm-evaluation-harness project and documentation (https://github.com/EleutherAI/lm-evaluation-harness); Hugging Face Open LLM Leaderboard retrospective (https://huggingface.co/spaces/open-llm-leaderboard/blog); LMSYS Chatbot Arena methodology and paper (https://lmsys.org/blog/2023-05-03-arena/, https://arxiv.org/abs/2403.04132); Zheng et al., "Judging LLM-as-a-Judge" (https://arxiv.org/abs/2306.05685); Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (https://arxiv.org/abs/2201.11903).

A Concrete Workflow: Reporting Effect Size Instead of (or Alongside) a P-Value

For a PM or analyst running internal model comparisons, the fix is not to abandon significance testing — p-values still have a role in flagging "this could plausibly be noise" — but to restructure the report so effect size and uncertainty come first, with the p-value demoted to a footnote. A workable five-step process:

1. Fix the measurement conditions before comparing anything. Hold prompt template, few-shot examples, decoding temperature, and max-token settings identical across models being compared. If this isn't possible (e.g., one model requires a different chat template), document the deviation explicitly rather than letting it silently inflate the apparent effect. Given the 1–5 point variance documented from formatting alone, this step alone eliminates a large share of spurious "wins."

2. Report the raw accuracy delta with a paired bootstrap confidence interval, not a p-value alone. Resample the item-level results (with replacement, same items for both models) 1,000–10,000 times, compute the accuracy delta each time, and report the 2.5th–97.5th percentile interval. This handles the non-independence problem (correlated errors across related questions) far better than a binomial test, and it directly answers "how big, with how much uncertainty" in one step.

3. Convert to a standardized effect size when comparing across benchmarks of different scales or baselines. If you're aggregating "wins" across GSM8K, MMLU, and HumanEval to make a portfolio-level claim, use Cohen's h or z-scored deltas so a 2-point gain on a benchmark with 95% baseline accuracy (where headroom is small) isn't averaged in as equivalent to a 2-point gain on a benchmark with 50% baseline accuracy.

4. Separate statistical significance from practical significance explicitly in the write-up. State both: "The accuracy delta is 2.3 points (95% CI: 0.4–4.1), which is statistically distinguishable from zero but falls within the range of variance we've observed from prompt-formatting changes alone, so we do not consider this a reliable basis for a model-swap decision without replication under at least two additional prompt configurations." This sentence pattern — effect, interval, and an explicit practical threshold — does more work than any single p-value.

5. Pre-register a minimum effect size that matters before running the comparison. Before looking at results, decide: what's the smallest accuracy gain that would justify the migration cost, re-testing effort, and risk of a model swap? If that threshold is 5 points and the observed effect (with interval) is 2.3 ± 1.9, the decision is "do not ship" regardless of whether p < 0.05. This single step — deciding the threshold before seeing the data — is the most effective defense against both p-hacking and its inverse, "effect-size-hacking," where a trivially small but statistically significant result is dressed up as a meaningful win because the threshold was never set.

Why This Matters More as Benchmarks Get Gamed

There's a second-order reason effect size matters beyond the statistics: as leaderboards become commercially important, benchmark scores become a target for optimization in ways that specifically exploit p-value thinking. A lab that knows evaluators will treat any statistically-significant delta as a meaningful "win" has an incentive to find benchmarks, prompt formats, or sampling configurations that produce small but significant deltas in their favor — which is cheap to do with enough benchmark iterations — rather than genuinely large capability improvements, which are hard and expensive. This is a close cousin of p-hacking in the clinical trial literature, and the research community has started naming it directly in discussions of benchmark contamination and leaderboard gaming. A PM who demands "effect size with interval, replicated across at least two measurement configurations" as the bar for a vendor claim makes this kind of gaming measurably harder to get away with, because small manufactured deltas tend to evaporate the moment you re-measure under slightly different conditions — which is exactly the instability a confidence interval is designed to expose.

Practical Takeaway for PMs and Analysts

If you remember one reframing from this piece, make it this: a p-value tells you whether a difference is probably not zero; an effect size with a confidence interval tells you whether the difference is big enough, and well enough established, to act on. For LLM evaluation specifically — where benchmarks are small relative to clinical norms, items are correlated, judges are themselves noisy measurement instruments, and prompt formatting alone can move scores several points — the gap between those two questions is unusually wide, and defaulting to p-value-only reporting will misfire in both directions: false confidence in noise-sized gains, and false dismissal of real gains that a small benchmark lacks the power to certify as "significant."

Concretely, before approving a model migration, a pricing-tier change, or a vendor comparison deck based on benchmark numbers, ask for three things: (1) the raw effect size with a bootstrap or paired confidence interval, not just a point estimate; (2) confirmation that prompt template, decoding settings, and sampling were held identical across the compared models; and (3) a pre-stated minimum effect size that would justify the decision, set before the numbers were known. If any of the three is missing, treat the reported "win" as provisional — not because the team is being dishonest, but because the standard reporting conventions in this field, including in headline vendor model-card comparisons, still routinely omit exactly the information needed to tell a real effect from a rounding error in benchmark noise.