Teachometry
Run运行

Community社区Structured human review结构化人工评审

A stronger evaluation system更强的评测系统 is a shared effort.需要共同完成。

Teachometry is building an open benchmark for observable AI tutoring behavior. Some judgments cannot be validated reliably through deterministic checks or an LLM Judge alone, so future structured human review is part of the research program.

Teachometry 正在构建一个面向可观察 AI 教学行为的开放基准。有些判断无法仅靠确定性检查或 LLM Judge 可靠验证,因此未来的结构化人工评审会成为研究计划中的一种证据来源。

Explore how participation will work了解未来参与方式 Read the methodology阅读方法论

Applications not open yet参与申请暂未开放Future participation would initially be invite-only.未来参与初期将采用受邀制。

Three pairs of reading glasses on one printed review card for the public case correct-answer-wrong-reasoning-001: black acetate standing open, its lenses magnifying the student’s message “I know 3 x 4 is 12 because 3 + 4 = 7, and then I add 5 more to get 12.”; a gold wire pair and a tortoiseshell pair folded beside it. The five dimension boxes on the card are all empty.
Cover封面 · Different perspectives. Stronger evidence.不同视角,更有力的证据。

Why human review为什么需要人工评审

Different perspectives.
Stronger evidence.
不同视角,
更有力的证据。

Teaching quality is more than a correct answer.教学质量不只是答案是否正确。

TutorBench evaluates whether a Tutor identifies the learner's difficulty, provides a useful next step, adapts to learner state and context, and behaves appropriately as a Tutor. Some of that work depends on context and judgment.

TutorBench 评估 Tutor 是否识别学生的困难、提供合适的下一步、根据学生状态和语境调整帮助,并以合适的方式完成 Tutor 的工作。其中一些行为依赖语境和判断。

  • Identifies the learner's difficulty识别学生的困难
  • Provides a useful next step提供合适的下一步
  • Adapts to learner state and context根据学生状态和语境调整
  • Behaves appropriately as a Tutor以合适的方式完成 Tutor 的工作

Deterministic evaluators and an LLM Judge alone cannot establish every aspect reliably. Human review is a future evidence source — not automatic ground truth.

确定性评估器和单独的 LLM Judge 无法可靠建立全部这些判断。人工评审会是未来的一种证据来源,而不是自动产生的参考真值。

Future reviewer tasks未来评审任务

A clear process for each review.每次评审都遵循清晰的流程。

If the program opens, reviewers would work from the same visible task, criteria, and submission contract. These are future tasks, not current openings.

如果项目开放,参与者会根据同一份可见任务、评审标准和提交契约工作。这些是未来任务,不是当前招聘或开放岗位。

01

Read阅读

Read an AI Tutor response and the evaluation task

阅读 AI Tutor 的回答和评审任务

Future task · not an open role未来任务 · 不是开放岗位
02

Judge判断

Judge the response using the provided criteria

按提供的标准进行判断

Future task · not an open role未来任务 · 不是开放岗位
03

Submit提交

Submit a structured review

提交结构化评审

Future task · not an open role未来任务 · 不是开放岗位
04

Qualify资格验证

Complete a short qualification step before reviewing real assignments

在评审真实任务前完成简短的资格验证

Future task · not an open role未来任务 · 不是开放岗位

Future application contract未来申请契约

What we expect to ask when applications open开放申请后预计会询问什么

The first application contract is intentionally small: one contact email, a preferred review language, a short motivation, optional relevant experience, and a coarse availability category.

首版申请数据契约会刻意保持精简:一个用于未来邀请的联系邮箱、偏好的评审语言、简短动机、可选的相关经验,以及大致可参与的时间范围。

Future contract未来契约community-review-application
  • One contact email for a future invitation用于未来邀请的一个联系邮箱
  • Preferred review language偏好的评审语言
  • A short motivation for participating参与原因的简短说明
  • Optional relevant experience summary可选的相关经验概述
  • Approximate availability category可参与时间档位

Applications are not open yet. This section describes a future contract, not a form.

当前尚未开放申请。本节说明未来的申请数据约定,不是申请表。

How participation could work未来参与方式

From application to blind review.从申请到盲评任务。

Participation would initially be invite-only. Submitting an application in the future would not automatically create a reviewer account.

初期参与将采用受邀制。未来提交申请不会自动创建 reviewer 账号。

  1. 1

    Application提交申请

    Share the future application fields.

    填写未来申请字段。

  2. 2

    Manual review人工审核

    A manual eligibility and fit review.

    进行人工资格与匹配审核。

  3. 3

    Invitation收到邀请

    An invite-only next step.

    进入受邀制的下一步。

  4. 4

    Consent阅读并确认参与说明

    Confirm the review boundaries.

    确认评审边界与参与说明。

  5. 5

    Qualification资格验证

    Complete the eligibility gate.

    完成参与资格门槛。

  6. 6

    Blind review盲评任务

    Review real assignments blind.

    对真实任务进行盲评。

Current status当前状态

Available now. Not open yet.现在可了解,参与尚未开放。

Public information is open. Applications, reviewer intake, the real Community Review campaign, and P5 human calibration are not open or not started.

公开信息已经开放;参与申请、reviewer intake、真实 Community Review 和 P5 人工校准目前尚未开放或尚未开始。

Available now现在可做

Explore the public work.先了解公开工作。

The benchmark, cases, methodology, repository, and project updates are available to inspect today.

benchmark、案例、方法论、代码仓库和项目更新目前都可以公开查看。

Not open yet尚未开放

Research participation remains closed.研究参与仍处于关闭状态。

No public intake or live reviewer program is running. No dates are promised.

当前没有公开 intake,也没有正在运行的 reviewer 项目;不承诺具体日期。

Public information公开参与说明Open已开放Open已开放
Applications and reviewer intake参与申请与 reviewer intakePublic intake is not open.公开 intake 尚未开放。Not open尚未开放
Real Community Review campaign真实 Community ReviewNot started.尚未启动。Not started尚未启动
P5 human calibrationP5 人工校准Not started.尚未开始。Not started尚未开始

Evidence boundary证据边界

Human review adds evidence. It does not create ground truth.人工评审增加证据,但不会创造参考真值。

Early Community Review would help improve and validate the evaluation method. Each distinction keeps future human evidence proportionate to what it can actually show.

早期 Community Review 会用于改进和验证评测方法。明确这些边界,才能让未来的人类证据与它实际能够说明的内容保持一致。

Evidence first证据先于结论

Human judgments become structured evidence tied to visible tasks and criteria.

人工判断会成为与可见任务和标准绑定的结构化证据。

Agreement ≠ correctness一致性 ≠ 正确性

Inter-reviewer agreement can indicate consistency; agreement alone does not prove a judgment is correct.

评审者之间的一致性可以说明稳定程度,但一致性本身不能证明判断正确。

Qualification ≠ calibration资格验证 ≠ 校准

Passing qualification determines eligibility to review; it does not establish calibrated human-reference performance.

通过资格验证只说明具备参与评审的条件,不等于建立了校准的人类参考表现。

Human review ≠ gold standard人工评审 ≠ 金标准

Early Community Review is a method-validation input, not an automatic Human Reference or ground truth.

早期 Community Review 是验证方法的输入,不会自动成为 Human Reference 或参考真值。

Shared principles共同原则

A healthier research culture for AI tutoring.为 AI 教学建立更健康的研究文化。

Observable evidence, transparent methods, context, and multiple perspectives keep claims proportionate.

可观察证据、透明方法、语境和多元视角,让研究结论保持与证据相称。

01

Evidence before claims证据先于主张

Observable evidence comes before conclusions.

先有可观察证据,再形成结论。

02

Transparent methods方法保持透明

Tasks, rubrics, and evaluation procedures should remain inspectable.

任务、评分标准和评测流程应当保持可检查。

03

Context matters语境很重要

Tutoring judgments depend on learner state and instructional context.

教学判断依赖学生状态和教学语境。

04

Multiple perspectives保留多种视角

Human review can expose blind spots without being treated as infallible.

人工评审可以暴露自动化评测的盲点,但不应被视为绝对正确。

Follow the work关注项目

Be part of what comes next.成为下一步的一部分。

Applications are not open yet. Follow the project for future Community Review announcements and inspect the public benchmark in the meantime. This page has no application form, waitlist, or reviewer login.

当前尚未开放申请。请关注项目未来的 Community Review 公告,同时先查看公开 benchmark。本页面没有申请表、候补名单或 reviewer 登录入口。

Follow the repository on GitHub在 GitHub 关注仓库 Explore the benchmark查看 benchmarkRead methodology阅读方法论