Read阅读
Read an AI Tutor response and the evaluation task
阅读 AI Tutor 的回答和评审任务
Future task · not an open role未来任务 · 不是开放岗位Teachometry is building an open benchmark for observable AI tutoring behavior. Some judgments cannot be validated reliably through deterministic checks or an LLM Judge alone, so future structured human review is part of the research program.
Teachometry 正在构建一个面向可观察 AI 教学行为的开放基准。有些判断无法仅靠确定性检查或 LLM Judge 可靠验证,因此未来的结构化人工评审会成为研究计划中的一种证据来源。
Explore how participation will work了解未来参与方式 Read the methodology阅读方法论
Applications not open yet参与申请暂未开放Future participation would initially be invite-only.未来参与初期将采用受邀制。
Why human review为什么需要人工评审
Different perspectives.
Stronger evidence.不同视角,
更有力的证据。
TutorBench evaluates whether a Tutor identifies the learner's difficulty, provides a useful next step, adapts to learner state and context, and behaves appropriately as a Tutor. Some of that work depends on context and judgment.
TutorBench 评估 Tutor 是否识别学生的困难、提供合适的下一步、根据学生状态和语境调整帮助,并以合适的方式完成 Tutor 的工作。其中一些行为依赖语境和判断。
Deterministic evaluators and an LLM Judge alone cannot establish every aspect reliably. Human review is a future evidence source — not automatic ground truth.
确定性评估器和单独的 LLM Judge 无法可靠建立全部这些判断。人工评审会是未来的一种证据来源,而不是自动产生的参考真值。
Future reviewer tasks未来评审任务
If the program opens, reviewers would work from the same visible task, criteria, and submission contract. These are future tasks, not current openings.
如果项目开放,参与者会根据同一份可见任务、评审标准和提交契约工作。这些是未来任务,不是当前招聘或开放岗位。
Read an AI Tutor response and the evaluation task
阅读 AI Tutor 的回答和评审任务
Future task · not an open role未来任务 · 不是开放岗位Judge the response using the provided criteria
按提供的标准进行判断
Future task · not an open role未来任务 · 不是开放岗位Submit a structured review
提交结构化评审
Future task · not an open role未来任务 · 不是开放岗位Complete a short qualification step before reviewing real assignments
在评审真实任务前完成简短的资格验证
Future task · not an open role未来任务 · 不是开放岗位Future application contract未来申请契约
The first application contract is intentionally small: one contact email, a preferred review language, a short motivation, optional relevant experience, and a coarse availability category.
首版申请数据契约会刻意保持精简:一个用于未来邀请的联系邮箱、偏好的评审语言、简短动机、可选的相关经验,以及大致可参与的时间范围。
community-review-applicationApplications are not open yet. This section describes a future contract, not a form.
当前尚未开放申请。本节说明未来的申请数据约定,不是申请表。
How participation could work未来参与方式
Participation would initially be invite-only. Submitting an application in the future would not automatically create a reviewer account.
初期参与将采用受邀制。未来提交申请不会自动创建 reviewer 账号。
Share the future application fields.
填写未来申请字段。
A manual eligibility and fit review.
进行人工资格与匹配审核。
An invite-only next step.
进入受邀制的下一步。
Confirm the review boundaries.
确认评审边界与参与说明。
Complete the eligibility gate.
完成参与资格门槛。
Review real assignments blind.
对真实任务进行盲评。
Current status当前状态
Public information is open. Applications, reviewer intake, the real Community Review campaign, and P5 human calibration are not open or not started.
公开信息已经开放;参与申请、reviewer intake、真实 Community Review 和 P5 人工校准目前尚未开放或尚未开始。
Available now现在可做
The benchmark, cases, methodology, repository, and project updates are available to inspect today.
benchmark、案例、方法论、代码仓库和项目更新目前都可以公开查看。
Not open yet尚未开放
No public intake or live reviewer program is running. No dates are promised.
当前没有公开 intake,也没有正在运行的 reviewer 项目;不承诺具体日期。
Evidence boundary证据边界
Early Community Review would help improve and validate the evaluation method. Each distinction keeps future human evidence proportionate to what it can actually show.
早期 Community Review 会用于改进和验证评测方法。明确这些边界,才能让未来的人类证据与它实际能够说明的内容保持一致。
Human judgments become structured evidence tied to visible tasks and criteria.
人工判断会成为与可见任务和标准绑定的结构化证据。
Inter-reviewer agreement can indicate consistency; agreement alone does not prove a judgment is correct.
评审者之间的一致性可以说明稳定程度,但一致性本身不能证明判断正确。
Passing qualification determines eligibility to review; it does not establish calibrated human-reference performance.
通过资格验证只说明具备参与评审的条件,不等于建立了校准的人类参考表现。
Early Community Review is a method-validation input, not an automatic Human Reference or ground truth.
早期 Community Review 是验证方法的输入,不会自动成为 Human Reference 或参考真值。
Shared principles共同原则
Observable evidence, transparent methods, context, and multiple perspectives keep claims proportionate.
可观察证据、透明方法、语境和多元视角,让研究结论保持与证据相称。
Observable evidence comes before conclusions.
先有可观察证据,再形成结论。
Tasks, rubrics, and evaluation procedures should remain inspectable.
任务、评分标准和评测流程应当保持可检查。
Tutoring judgments depend on learner state and instructional context.
教学判断依赖学生状态和教学语境。
Human review can expose blind spots without being treated as infallible.
人工评审可以暴露自动化评测的盲点,但不应被视为绝对正确。
Follow the work关注项目
Applications are not open yet. Follow the project for future Community Review announcements and inspect the public benchmark in the meantime. This page has no application form, waitlist, or reviewer login.
当前尚未开放申请。请关注项目未来的 Community Review 公告,同时先查看公开 benchmark。本页面没有申请表、候补名单或 reviewer 登录入口。
Follow the repository on GitHub在 GitHub 关注仓库 Explore the benchmark查看 benchmarkRead methodology阅读方法论