The Illusion of Improvement

Model releases are announced with rising benchmark scores, yet practitioners routinely report that "upgraded" models feel worse in production: longer answers, more boilerplate, ignored formatting instructions, and answers that flip when a question is rephrased. We call this gap between measured capability and delivered behavior the illusion of improvement. We hypothesize that optimization pressure toward public benchmarks and preference-tuned reward signals systematically trades behavioral fidelity — instruction obedience, brevity, and consistency — for capability-benchmark gains.

To test this, we introduce llm-diff, an open-source behavioral regression harness, and apply it across [N] successive model releases from [K] model families. Placeholder headline finding, e.g. "across X of Y release pairs, benchmark deltas and behavioral-fidelity deltas moved in opposite directions." We argue that release engineering for LLMs needs behavioral regression testing as a first-class gate, alongside benchmarks.

• The upgrade ritual: teams swap model versions on the strength of release notes and leaderboard deltas, with no behavioral equivalent of a regression suite.
• Anecdotal regression classes reported by practitioners: verbosity creep, filler preambles, markdown scaffolding by default, formatting-instruction drift, refusal creep, and phrasing-sensitive answers.
• Goodhart's law framing: when a benchmark becomes a target, it stops measuring what deployers care about; behavior is what users actually receive.
• Contribution list: (1) a definition and taxonomy of behavioral fidelity; (2) llm-diff, a reproducible open-source measurement harness; (3) an empirical study across successive releases PENDING; (4) a proposed release-gating practice ("behavioral changelogs").

3.1  The llm-diff framework

• Probe → score → diff pipeline; deterministic decoding (temperature 0); candidate−baseline deltas; exit-code gating for CI; JSON reports for aggregation. All probes and scorers are open source and versioned.

3.2  Behavioral dimensions and scoring

• Planned extensions for the full study: Hallucination Propensity, Refusal Sensitivity, Domain Retention, Sycophancy Drift.
• Explicit note on construct validity: scorers are transparent heuristics; each dimension reports raw sub-metrics so readers can audit the mapping from text to number.

3.3  Probe battery construction

• Scale from the 4-probe MVP smoke test to target ≥50 probes per dimension, stratified by difficulty and phrasing.
• Paraphrase frames generated per probe to measure phrasing sensitivity; nonsense-vocabulary templates to resist training-data memorization.
• Community probe packs with review criteria Process TBD.

3.4  Models and release pairs

• Within-family successive releases at matched parameter scale, e.g. Llama 3 → 3.1 (8B), Qwen2.5 → Qwen3 (7–8B), Mistral 7B v0.1 → v0.3, Gemma 2 → Gemma 3, plus closed-model API version pairs Final list TBD.
• Serving via local Ollama (pinned quantizations) and Hugging Face Inference Providers' OpenAI-compatible router; provider and quantization recorded per run as confounders.

3.5  Statistical treatment

• [R] repeated runs per probe; report means with bootstrap confidence intervals; pre-registered thresholds for “regression” vs “noise”; correlation of behavioral deltas against published benchmark deltas.

• Table 1: Per-dimension deltas for each release pair, with confidence intervals. PENDING
• Table 2: Correlation between announced benchmark improvements and measured behavioral-fidelity change. PENDING
• Case studies: the largest observed regression and the cleanest observed improvement, with raw response excerpts. PENDING

• Interpreting the fidelity/benchmark trade-off (if observed): optimization pressure, preference-model length bias, safety-tuning side effects.
• Why arena-style human preference can reward the exact behaviors deployers experience as regressions.
• What “improvement” should mean for a release: capability and behavioral changelog.

• Heuristic scorers (keyword preamble detection, regex verdict extraction) measure proxies, not ground truth; all sub-metrics are published for audit.
• English-only, single-turn probes in this study; multilingual and multi-turn behavior is future work.
• MVP battery is small; statistical power arrives with the expanded probe set.
• Serving-stack confounders (quantization, provider kernels, sampling implementations) can masquerade as model behavior; we record but cannot fully eliminate them.
• Temperature-0 decoding is not bitwise deterministic on all backends.

Behavioral regressions are invisible to benchmarks by construction and are currently discovered by end users in production. We call on the industry to adopt behavioral regression testing as a release gate: publish behavioral changelogs alongside benchmark tables, run pairwise diffs in CI before every model swap, and contribute domain probe packs to open harnesses.

All probes, scorers, raw responses, and analysis scripts will be released at github.com/Pluto-AI-Labs/llm-diff under MIT. Artifact DOI pending

To be completed — slots reserved for: benchmark papers (MMLU, HELM, GPQA); arena/preference evaluation and length-bias studies; RLHF sycophancy and verbosity-bias literature; CheckList and behavioral-testing work; API model-drift studies; Goodhart's law in ML.