CI gating

Fail PRs on quality regressions. ReasonRank’s recommendation engine only surfaces a cheaper model when a paired cluster bootstrap shows it is non-inferior to the baseline (QUALITY_TOLERANCE = 0.02). The CLI applies the same gate so a PR that changes a prompt, skill, or model routing cannot merge if quality slips.

Prerequisites

1. An agent (suite) with ≥5 shared test cases.
2. Model configs for the baseline and candidate(s).
3. A PAT (Settings → API Keys) with runs:write and runs:read.
4. Provider keys / HTTP endpoint / external harness configured so the run can actually execute.

Local

export REASONRANK_API_KEY=rr_pat_…
pnpm --filter @reasonrank/cli exec tsx src/cli.ts run \
  --agent "$AGENT_ID" \
  --models "$BASELINE_ID,$CANDIDATE_ID" \
  --wait

pnpm --filter @reasonrank/cli exec tsx src/cli.ts assert \
  --run "$RUN_ID" \
  --baseline "$BASELINE_ID"

Exit code 1 means: insufficient shared cases, or the 95% CI lower bound on (candidate − baseline) quality is below -0.02.

GitHub Actions

# .github/workflows/reasonrank.yml
name: ReasonRank gate
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'agents/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - uses: ./.github/actions/reasonrank-gate
        with:
          api_key: ${{ secrets.REASONRANK_API_KEY }}
          agent_id: ${{ vars.REASONRANK_AGENT_ID }}
          model_ids: ${{ vars.REASONRANK_BASELINE_ID }},${{ vars.REASONRANK_CANDIDATE_ID }}
          baseline_id: ${{ vars.REASONRANK_BASELINE_ID }}

What “fail” means

This is intentionally the same bar as in-product recommendations — CI and the Savings page never disagree on the math.