Documentation — llm-diff Quick Start

Installation Local Testing (Ollama)Cloud Testing (Groq)Paid Cloud (OpenAI)CLI Flags & Options CI/CD Integration What's Next?

Step 1

Installation

llm-diff requires Python 3.10+. The package is published on PyPI under pluto-llm-diff, which exposes the llm-diff terminal executable.

bash

pip install pluto-llm-diff

Verify your installation:

bash

llm-diff --help

Step 2

Local Testing (100% Free & Offline via Ollama)

Run behavioral diffs entirely offline on your local machine using Ollama.

bash

llm-diff ollama/qwen2.5:0.5b ollama/llama3.2:1b

Tip: If a model spec starts with the ollama/ prefix, llm-diff auto-routes to the Ollama backend. Explicitly setting --backend ollama is optional when using prefixed model specs.

Step 3

Cloud Testing (Free via Groq)

Benchmark massive open-weight models (Llama 3.1 vs 3.3) for free using Groq's high-speed OpenAI-compatible API.

Get a free API key from console.groq.com.
Point --openai-base-url to Groq's endpoint.

bash

llm-diff openai/llama-3.1-70b-versatile openai/llama-3.3-70b-versatile \
  --backend openai \
  --openai-base-url https://api.groq.com/openai/v1 \
  --api-key your_groq_api_key_here

Step 4

Paid Cloud Testing (OpenAI)

To compare standard OpenAI models (e.g., gpt-4o-mini vs gpt-4o), export your OPENAI_API_KEY and run:

bash

export OPENAI_API_KEY="sk-proj-..."

llm-diff openai/gpt-4o-mini openai/gpt-4o

Reference

CLI Flags & Options

--backend

Fallback backend when the model spec lacks a prefix (ollama or openai). Default: ollama.

--ollama-url

Base URL for the local Ollama API server. Default: http://localhost:11434 (or OLLAMA_HOST).

--openai-base-url

Base URL for OpenAI-compatible servers. Default: https://api.openai.com/v1 (or OPENAI_BASE_URL).

--api-key

API key for OpenAI-compatible endpoints (or set OPENAI_API_KEY).

--json

Emit a machine-readable JSON report instead of formatted Rich terminal tables.

--show-responses

Print full raw model responses alongside evaluation scores for debugging.

--temperature

Sampling temperature for generation. Default: 0.0 for maximum reproducibility.

--max-tokens

Maximum output tokens per probe evaluation. Default: 512.

--timeout

Per-request API call timeout in seconds. Default: 120.0.

--retries

Retry attempts on transient network/API failures (timeouts, 429s, 5xx). Default: 2.

Automation

CI/CD Integration & Automation

llm-diff is designed for automated regression testing. Use --json to emit a structured report containing all probe outputs, metric deltas, and execution metadata.

Exit Codes

exit 0Success

All probe evaluations completed successfully across both models.

exit 1Fatal Error

Unrecoverable error: invalid arguments, missing API key, or bad base URL.

exit 3Partial Failure

Diff completed, but one or more probes timed out or errored (ERR rows).

GitHub Actions Example

Save behavioral diff reports as pipeline artifacts:

.github/workflows/llm-diff.yml

name: LLM Behavioral Regression Test

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  behavior-diff:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python 3.10
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Install llm-diff
        run: |
          python -m pip install --upgrade pip
          pip install pluto-llm-diff

      - name: Run behavioral diff report
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          llm-diff openai/gpt-4o-mini openai/gpt-4o --json > diff-report.json

      - name: Upload diff report artifact
        uses: actions/upload-artifact@v4
        with:
          name: llm-diff-report
          path: diff-report.json

Keep going