git diff, but for
LLM behavior.
Stop vibe-testing your model upgrades. Probe two language models with behavioral tests, score their responses, and catch silent regressions before you deploy.
The Illusion of Improvement
Every week, a new LLM drops. Benchmarks claim higher intelligence, but in production, JSON schemas break, outputs grow 40% longer, and edge-case prompts get rejected. Standard benchmarks miss real-world behavioral regressions.
Trusting release notes blindly
Marketing claims 'better instruction following'. Production says otherwise.
Running full evaluations
MMLU, HumanEval, GPQA. Days of compute for numbers that don't match your workload.
Manual vibe testing
Eyeball 10 prompts, ship to prod, wait for the Slack fire drill.
Behavioral Regression Testing for LLMs
Three orthogonal probes. Deterministic scoring. Machine-readable output. Everything you need to know if the new model actually works for your workload.
Instruction Fidelity
Probes strict formatting constraints and multi-turn instructions. Scores compliance from 0.0 to 1.0.
Verbosity Profile
Tracks output explosion and fluff. Measures token count, filler phrases, and unnecessary headers.
Reasoning Consistency
Tests logic stability under variable phrasing. Sends isomorphic syllogisms and checks verdict alignment.
Install in seconds. Test in seconds.
Works with any OpenAI-compatible endpoint. Zero config for local Ollama models. One flag to swap providers.
# Install from PyPI
pip install pluto-llm-diff
# Verify installation
llm-diff --helpBuilt for CI/CD.
Emit machine-readable JSON and use standardized exit codes to fail your GitHub Actions or Jenkins pipelines automatically when a model regresses.
- name: LLM regression gate
run: |
llm-diff \
openai/gpt-4o-mini \
openai/gpt-4o \
--json > report.json
# exits non-zero if any probe regresses
- uses: actions/upload-artifact@v4
with:
name: llm-diff-report
path: report.json