Teachometry
Run运行

Our methodology我们的方法论TutorBenchtutor-benchmark

How do you measure teaching?如何测量教学?

Teachometry evaluates observable tutoring behavior in structured authored benchmark cases using transparent and reproducible evaluation procedures.

Teachometry 使用透明且可复现的评测流程,在结构化、人工编写的基准案例中评估可观察的教学行为。

Steel gauge blocks on a black granite surface plate; four of them are wrung together into one column.
Cover封面 · Four steel blocks wrung into one length: 50 + 20 + 1.4 + 1.005 = 72.405 mm.四块钢制量块研合成一段长度:50 + 20 + 1.4 + 1.005 = 72.405 毫米。

§1The evaluation pipeline评测流程

From a structured case to a benchmark result.

从结构化案例走向基准结果。

  1. 01Structured case结构化案例Authored scenario with learner profile, goal, and context.包含学生画像、目标与语境的人工编写情境。
  2. 02Tutor responseTutor 回复The response produced for the learner’s structured case.针对结构化学生案例生成的回复。
  3. 03Evaluator evidence评估器证据Evidence from deterministic evaluators and the Semantic Judge.来自确定性评估器和 Semantic Judge 的证据。
  4. 04Atomic rubrics原子化评分标准Fine-grained criteria with explicit scoring levels.带有明确评分等级的细粒度标准。
  5. 05Benchmark result基准结果Category aggregation with documented, comparable outputs.以有记录、可比较的输出进行类别聚合。
Fig. 1图 1The five stages, from a structured case to a benchmark result.五个阶段:从结构化案例到基准结果。

§2Why we evaluate this way为什么这样评测

We start from the mathematics, not from a count of prompts.我们从数学出发,而不是从题目数量出发。

A tutor can give a correct answer and still teach badly. So a case does not begin with a question to answer. It begins with an idea a learner can misunderstand, a moment where that misunderstanding shows, and a decision a good tutor has to make next. In mathematics we keep those three things apart, and linked: a knowledge node, a teaching case, and an evaluation decision.

导师可以给出正确答案,却仍然教得不好。所以一个案例不是从“出一道题”开始,而是从三件事开始:学习者可能误解的一个概念,这个误解显露出来的那一刻,以及好导师接下来必须做出的判断。在数学上,我们把这三层分开记录,又彼此相连:知识节点、教学案例、评测决策。

A white card price tag tied with violet twine, a violet pencil beside it. On the tag in pencil: 200, plus 20 %, minus 20 %, equals, and an empty line.
Fig. 2图 2The learner’s own sum from the case below. The last price stays blank: in this case the tutor may give a hint, but must not write the answer in.下面这个案例里,学习者自己写下的算式。最后的价格留空:在这个案例中,导师可以提示,但不能替学习者写出答案。

Here is that one case, followed through all three layers. Every field is quoted from the repository’s experimental mathematics suite.

下面用这一个案例,把三层完整走一遍。每一项内容都引自代码库中的实验性数学案例集。

  1. 01Knowledge node知识节点

    Numbers, quantities and proportional reasoning数与数量关系

    Percent as a fraction of a base百分数及其参照量percent-base Successive multiplicative changes连续百分比变化successive-changes

    Common misconception常见误区
    Assuming successive equal percentage changes cancel认为相等的连续百分比变化会相互抵消
    Status性质
    A prompt for writing cases, never a diagnosis of a real learner. The link between the two nodes is an author hypothesis.这是编写案例的提示,不是对真实学习者的诊断。两个节点之间的联系是作者提出的假设。
  2. 02Teaching case教学案例

    mc-percent-reversal

    A product costs 200. It goes up 20% and then down 20%. I said it must be back to 200 because the changes cancel. I can calculate 20% of 200, so what is wrong with my reasoning?一件商品原价 200,先涨 20%,再降 20%。我说它一定回到 200,因为两次变化抵消了。我会算 200 的 20%,那我的推理错在哪里?

    The tutor sees导师看到

    Known concept已掌握的概念
    percent of a number求一个数的百分之几
    Help allowed允许的帮助
    Hints only只能提示

    Only the evaluator holds只有评测方掌握

    The misconception误区
    Treats equal percentage changes applied to different bases as cancelling additively.把作用在不同基数上的相等百分比变化,当作可以相加抵消。
    The reference参考答案
    200 → 240 → 192, a net 4% decrease200 → 240 → 192,净降 4%
  3. 03Evaluation decision评测决策

    offer_conceptual_hint

    Offer a conceptual hint.给出概念性提示。

    • Correctness正确性Identify that the second 20% refers to the increased price and cannot cancel the first 20% change.指出第二个 20% 针对的是涨价后的价格,因此不能抵消第一次的 20% 变化。
    • Guidance引导能力Provide a concise executable next step to determine the intermediate price then use it as the second percentage base without revealing the final price.给出一个简短、可执行的下一步:先求出中间价格,再把它作为第二次百分比的基数;不透露最终价格。

    A full worked solution would be correct mathematics and would still miss this case. The learner can already do the arithmetic, asked what is wrong with the reasoning, and is allowed hints only.

    直接给出完整解答,数学上没有错,却仍然不符合这个案例的要求:学习者已经会计算,问的是推理错在哪里,而且这个案例只允许提示。

Fig. 3图 3One case, three layers. The dashed line is the response boundary: the tutor never sees the misconception label or the reference answer. Chinese text is our translation.一个案例,三层结构。虚线是回复边界:导师看不到误区标签和参考答案。中文为本站译文。

The map behind the cases案例背后的知识图谱

The knowledge map arranges school mathematics, and a first slice of introductory university topics, into 8 domains and 44 nodes. Each case is linked only to the nodes it actually exercises, so the map also shows plainly where no case exists yet. Those gaps are kept visible on purpose: we would rather add a case where the mathematics needs one than add cases to fill a grid.

知识图谱把中小学数学,以及一小部分大学入门内容,整理成 8 个领域、44 个节点。每个案例只关联它真正考查的节点,所以图谱也清楚地显示哪里还没有案例。这些空白是有意保留的:我们宁愿在数学上确实需要的地方补案例,也不为了填满表格而凑数。

8 domains · 44 nodes · 61 candidate relations · 20 cases on 16 task structures8 个领域 · 44 个节点 · 61 条候选关系 · 20 个案例,16 种任务结构

Quantity数量

  1. Numbers, quantities and proportional reasoning数与数量关系4 / 7

    • Whole and unit reference整体与单位基准 (no case yet)(暂无案例)
    • Integer operations整数与运算规则 (no case yet)(暂无案例)
    • Fractional units and denominators分数单位与分母 (linked to a case)(已有案例)
    • Equivalent fractions and shared units分数等值与通分 (linked to a case)(已有案例)
    • Ratios and proportional relationships比、比例与倍数关系 (no case yet)(暂无案例)
    • Percent as a fraction of a base百分数及其参照量 (linked to a case)(已有案例)
    • Successive multiplicative changes连续百分比变化 (linked to a case)(已有案例)

Change and relationships变化与关系

  1. Algebra and equations代数与方程3 / 7

    • Variables, expressions and equivalence变量、代数式与等式 (no case yet)(暂无案例)
    • Linear equations and inverse operations一元一次方程与等价变形 (no case yet)(暂无案例)
    • Expansion and factorization整式展开与因式分解 (no case yet)(暂无案例)
    • Rational expressions and excluded values分式及定义域限制 (linked to a case)(已有案例)
    • Inequality transformations and signs不等式变形与符号 (no case yet)(暂无案例)
    • Quadratic roots and discriminant二次方程与判别式 (linked to a case)(已有案例)
    • Positive and negative root conditions二次方程根的正负条件 (linked to a case)(已有案例)
  2. Functions and change函数与变化2 / 7

    • Coordinate graphs坐标系与图像读数 (no case yet)(暂无案例)
    • Functions as mappings and domains函数概念、定义域与对应关系 (no case yet)(暂无案例)
    • Function composition and domain restrictions复合函数与定义域 (linked to a case)(已有案例)
    • Inverse functions and one-to-one restrictions反函数与单射限制 (no case yet)(暂无案例)
    • Exponential expressions and growth指数运算与指数变化 (no case yet)(暂无案例)
    • Logarithm domain, base and inequality direction对数不等式与底数单调性 (linked to a case)(已有案例)
    • Limits and continuity极限与连续性 (no case yet)(暂无案例)

Space and shape空间与形状

  1. Geometry and spatial reasoning几何与空间推理2 / 6

    • Measurements of length and area长度、面积与度量单位 (no case yet)(暂无案例)
    • Angles and triangle properties角与三角形性质 (no case yet)(暂无案例)
    • Similarity, length and area scale相似与比例缩放 (linked to a case)(已有案例)
    • Right triangle and distance勾股定理及使用条件 (no case yet)(暂无案例)
    • Lines, circles and coordinate constraints解析几何及直线圆关系 (linked to a case)(已有案例)
    • Trigonometric definitions and geometry三角比、三角函数与几何关系 (no case yet)(暂无案例)

Uncertainty and data不确定性与数据

  1. Counting and probability计数与概率4 / 5

    • Multiplication, permutations and unordered choices分类、分步计数原理 (linked to a case)(已有案例)
    • Combinations and counting without order组合与不计顺序的选取 (linked to a case)(已有案例)
    • Events and equally likely sample spaces随机事件与样本空间 (no case yet)(暂无案例)
    • Conditional probability and restricted sample spaces条件概率与样本空间限制 (linked to a case)(已有案例)
    • Base rates and inverse conditional probability基准率与逆向条件概率 (linked to a case)(已有案例)
  2. Statistics and data interpretation统计与数据解释1 / 4

    • Weighted averages and aggregation统计汇总与加权平均 (no case yet)(暂无案例)
    • Subgroup rates and Simpson's paradox分层统计与辛普森悖论 (linked to a case)(已有案例)
    • Variation and distributions离散程度与数据分布 (no case yet)(暂无案例)
    • Sampling and inference limitations抽样、统计推断与因果区分 (no case yet)(暂无案例)

TutorBench’s own domainsTutorBench 自设领域

  1. Proof, logic and generalization证明、逻辑与推广4 / 4

    • Sequences and recurrences数列与递推关系 (linked to a case)(已有案例)
    • Quantifiers, cases and counterexamples逻辑量词、命题与反例 (linked to a case)(已有案例)
    • Induction base and step数学归纳法 (linked to a case)(已有案例)
    • Inequality proof and equality cases不等式证明及等号条件 (linked to a case)(已有案例)
  2. Introductory calculus and linear algebra微积分与线性代数入门2 / 4

    • Derivatives and local rates of change导数与瞬时变化率 (no case yet)(暂无案例)
    • Optimization and domain endpoints闭区间优化与边界分析 (linked to a case)(已有案例)
    • Linear systems and consistency线性方程组与相容性 (no case yet)(暂无案例)
    • Rank, dependence and consistency矩阵秩、线性相关与解空间 (linked to a case)(已有案例)
Fig. 4图 422 of 44 nodes are linked to at least one authored case. This is case-design coverage: it shows where cases exist, not how any tutor performed on them. The domains are grouped as the map groups them: PISA’s four content areas, then TutorBench’s own.44 个节点中有 22 个关联了至少一个案例。这是案例设计的覆盖情况:它说明哪里有案例,不代表任何导师在这些知识上的表现。领域按图谱自身的分组排列:先是 PISA 的四类数学内容,再是 TutorBench 自设的领域。

What the map is not这张图谱不是什么

  • Not a required order. The 61 links between nodes are author hypotheses about what helps, not proven prerequisites. Some ideas have more than one route: conditional probability can be reached by counting outcomes directly, or through combinations.不是必经的学习顺序。节点之间的 61 条联系,是作者对“什么有帮助”的假设,不是已证实的先修条件。有些概念可以从不同路径到达:条件概率既可以直接数出所有结果,也可以借助组合方法。
  • Not a difficulty scale. Stage labels such as lower or upper secondary are rough contexts for writing cases, not grades and not a measured difficulty.不是难度等级。“初中”“高中”等学段标签只是编写案例时的大致背景,不是年级,也不是测出来的难度。
  • Not a result. The 20 mathematics cases are an experimental suite outside the public benchmark score. Nothing here reports how any tutor did on them.不是评测结果。这 20 个数学案例属于实验性案例集,不计入公开基准分数。这里没有报告任何导师在这些案例上的表现。
  • Not certified curriculum coverage. Links to national curricula and exams are still being reviewed clause by clause.不是经认证的课标覆盖。与各国课程标准和考试的对应关系,仍在逐条审核中。

§3Five evaluation dimensions五个评测维度

Five complementary lenses on tutoring behavior.五个互补的教学观察视角。

We evaluate AI tutors across the five score dimensions in the public benchmark artifact, capturing complementary aspects of observable tutoring behavior in structured cases.

我们通过公开基准产物中的五个评分维度评估 AI Tutor,覆盖结构化案例中可观察教学行为的不同互补侧面。

  1. Dimension 01维度 01 · Score lens评分视角 Correctness正确性 Whether the Tutor stays factually and conceptually correct.Tutor 是否在事实与概念层面保持正确。 What we look for我们关注什么Factual and conceptual accuracy事实与概念准确性
  2. Dimension 02维度 02 · Score lens评分视角 Diagnosis诊断能力 Whether it identifies the learner’s actual error, gap, or reasoning issue.是否识别出学生真实的错误、知识缺口或推理问题。 What we look for我们关注什么Identifies learner needs and misconceptions识别学生需求与错误概念
  3. Dimension 03维度 03 · Score lens评分视角 Guidance引导能力 Whether its explanation or hint helps the learner make progress.解释或提示是否帮助学生取得进展。 What we look for我们关注什么Provides clear, correct, productive guidance提供清晰、正确且能推动学习的引导
  4. Dimension 04维度 04 · Score lens评分视角 Adaptation适应能力 Whether it changes its help for the learner’s state and context.是否根据学生当前状态与语境调整帮助方式。 What we look for我们关注什么Adapts to learner state, level, and context根据学生状态、水平与语境进行适应
  5. Dimension 05维度 05 · Score lens评分视角 Actionability可执行性 Whether it leaves the learner with a clear, executable next step.是否让学生得到清晰且可执行的下一步。 What we look for我们关注什么Actionable next steps for the learner为学生提供可执行的下一步
Fig. 5图 5An illustration of complementary lenses, not an evaluator order or a score.这是互补视角的示意,不代表评估器执行顺序,也不是评分。
Different aspects. A more complete picture.不同侧面。形成更完整的图景。

§4From response to result从回复到结果

A transparent evaluation architecture.透明的评测架构。

Our evaluation framework combines atomic rubrics, explicit evidence boundaries, and multiple evaluators to support reproducible and auditable results.

我们的评测框架结合原子化评分标准、明确的证据边界和多个评估器,以支持可复现、可审计的结果。

Tutor-visible input ends at the response boundary. Evaluator-only annotations stay on the evaluator side.Tutor 可见输入止于回复边界。仅供评估器使用的注释始终保留在评估器一侧。

  1. Structured case结构化案例Authored scenario with learner context人工编写情境,包含学生语境
  2. Tutor responseTutor 回复Tutor-visible interactionTutor 可见交互
  3. Evaluator evidence评估器证据 Deterministic evaluators确定性评估器observable proxy checks可观察代理检查Semantic JudgeSemantic Judgeevaluator boundary评估器边界
  4. Atomic rubrics原子化评分标准Criteria and scoring levels标准与评分等级
  5. Category aggregation类别聚合Five score dimensions五个评分维度
  6. Benchmark result基准结果Structured, comparable output结构化、可比较输出
Fig. 6图 6Transparent, reproducible, and auditable. A Judge is an evaluator boundary, not ground truth; unresolved Judge-required evidence remains unresolved and cannot silently become a valid score.透明、可复现、可审计。Judge 是评估器边界,不是参考真值;需要 Judge 而尚未解决的证据必须保持未解决,不能静默变成有效分数。

§5What we measure我们测量什么

What we measure我们测量什么

Observable tutoring behavior.可观察的教学行为。

We focus on what the AI Tutor does and says in the interaction, evaluated through structured rubrics and observable evidence.

我们关注 AI Tutor 在交互中实际做了什么、说了什么,并通过结构化评分标准和可观察证据进行评估。

  • Correctness正确性Factual and conceptual accuracy事实与概念准确性
  • Diagnosis诊断能力Identifies learner needs and misconceptions识别学生需求与错误概念
  • Guidance引导能力Provides clear, correct, productive guidance提供清晰、正确且能推动学习的引导
  • Adaptation适应能力Adapts to learner state, level, and context根据学生状态、水平与语境进行适应
  • Actionability可执行性Actionable next steps for the learner为学生提供可执行的下一步

Case-conditioned evidence, not a claim of general teaching effectiveness.

这是基于具体案例条件的证据,不是对一般教学有效性的主张。

What we do not measure我们不测量什么

Learning outcomes (not yet).学习结果(暂时不测)。

We currently do not measure long-term learning gains or real-world educational outcomes. These require longitudinal studies beyond the scope of this benchmark.

我们目前不测量长期学习增益或真实世界教育结果。这些问题需要超出本基准范围的纵向研究。

  • Actual long-term learning真实长期学习
  • Knowledge retention over time随时间保持的知识留存
  • Transfer to new contexts迁移到新情境
  • Student satisfaction学生满意度
  • Real classroom outcomes真实课堂结果

§6Methodology status方法论状态

A research effort in progress.一项仍在进行的研究工作。

We are building a rigorous and open evaluation infrastructure. Some components are available now, while others will be developed through future work.

我们正在构建严格且开放的评测基础设施。其中一些组件已经可用,另一些仍将在后续工作中完善。

Table 1表 1Current methodology status当前方法论状态
Component组件Status状态Notes说明
Calibration infrastructure校准基础设施Available可用Provider-independent contracts and synthetic pipeline fixtures are available.不依赖服务提供方的契约和合成流程 fixture 已可用。
Community Review infrastructureCommunity Review 基础设施Private staging ready私有 staging 已就绪Public launch remains blocked; reviewer intake remains closed, and the real Community Review campaign has not started.公开发布仍被阻塞;reviewer intake 仍关闭,真实 Community Review 活动尚未启动。
Human calibration (P5)人工校准(P5)Not started尚未开始No real human calibration data is claimed.当前不主张存在真实人工校准数据。
Judge-vs-human validationJudge 与人工对照验证Not completed尚未完成A Judge result is not presented as human-reference validation.Judge 结果不会被呈现为人工参考验证。
Statistical validation统计验证Not completed尚未完成No statistical validation claim is made for the current benchmark.当前基准不主张已经完成统计验证。

§7Our commitment我们的承诺

Transparent enough to challenge.
Structured enough to reproduce.
透明到足以被质疑。
结构清晰,便于复现。

Our methodology, rubrics, and benchmark data are openly available for inspection, feedback, and reuse. We invite the community to evaluate, critique, and build on this work.

我们公开方法论、评分标准和基准数据,供检查、反馈与复用。欢迎社区评估、质疑并在此基础上继续建设。