Run the Benchmark — Tutor Benchmark

Quickstart

Run the provider-free demo

Quickstart needs no API key, Judge, or network connection. It runs four fixed cases from tutor-eval-v0.1@0.1, an existing development/smoke subset, and reports deterministic checks without an official score.

git clone https://github.com/shuangyan123/tutorbench.git
cd tutorbench
npm ci
npm run quickstart

# Published v0.1.0 package:
npm install tutor-benchmark
tutorbench quickstart

Full benchmark

Run the canonical evaluation path

npm run benchmark remains the full local benchmark for tutor-eval-v0.2a@0.2a.6. Its semantic boundary includes Judge-required rubrics. Without an explicitly configured Judge, those criteria remain unresolved and the normal run reports errors with no score; Quickstart does not replace or weaken that behavior.

npm run benchmark

Use any language

Connect an external Tutor over HTTP

Any runtime that accepts JSON and serves POST /respond can implement the Tutor boundary. The adapter sends Tutor-visible input only and keeps Judge evidence on the evaluator side.

python examples/http-python-tutor/server.py

# Published package:
tutorbench run \
  --http http://127.0.0.1:8000/respond \
  --limit 3

# From a clone after npm run build:
node dist/src/cli/tutorbench.js run \
  --http http://127.0.0.1:8000/respond \
  --limit 3

Real-model evidence

Separate Product Tutor and canonical model evidence

The Product path freezes TutorTurnInput responses without a generation spec. The canonical model path freezes exact execution-packet responses with a generation spec. Both keep failed case/runs in a sanitized report and replay offline; neither discovers credentials, retries calls, or writes website public data.

tutorbench collect \
  --http http://127.0.0.1:8000/respond \
  --provider <provider> \
  --model <actual-model-id> \
  --prompt-version product-config-v3 \
  --provenance external \
  --limit 3 \
  --output artifacts/product/product.json

tutorbench collect-model \
  --http http://127.0.0.1:9000/generate \
  --provider <provider> \
  --model <actual-model-id> \
  --limit 3 \
  --output artifacts/real-model/model.json

tutorbench evaluate \
  --corpus artifacts/real-model/model.json

Canonical execution mode

Freeze the benchmark conditions first

Export a TutorExecutionPacket with the versioned TutorGenerationSpec, exact prompt identity, canonical messages, and output cap. The default baseline-native-default profile leaves optional temperature, reasoning, and seed controls unconstrained so provider-native behavior is not misrepresented as identical across vendors.

npm run tutor:export-execution -- -- --case fraction-misconception-001
npm run tutor:export-cases
npm run tutor:corpus:validate -- -- --corpus path/to/corpus.json
npm run benchmark:corpus -- -- --corpus path/to/corpus.json

Controlled optional generation parameters: none

tutor:export-cases is the semantic Tutor-visible adapter packet. tutor:export-execution is the canonical benchmark packet used to make model runs comparable. Neither packet includes evaluator-only annotations. The same benchmark does not imply that every provider exposes identical inference knobs.

Minimal adapter shape

Keep the provider at the edge

The adapter receives a typed, Tutor-visible input and returns a text response. Provider metadata stays outside the core benchmark result contract.

import type { TutorUnderTest } from "./src/contracts/tutor.js";

const tutor: TutorUnderTest = {
  id: "my-tutor",
  async respond(input) {
    return {
      text: await myTutor(input.currentStudentMessage),
    };
  },
};

Optional Judge path

Explicit, offline by default

The repository has separate opt-in OpenAI Responses and DeepSeek Chat Completions Judge providers. Dry-run/request tests stay offline; live execution requires explicit local configuration. The website build never calls a Judge provider and the browser never receives credentials.

npm run judge:openai -- -- --dry-run

# Frozen-corpus DeepSeek Judge subset:
node dist/src/cli/tutorbench.js evaluate \
  --corpus artifacts/real-model/baseline.json \
  --limit 1 \
  --judge-deepseek