In blind A/B comparison, raters prefer personalised (Adaptive) questions over non-personalised (Random-topic) questions about 68–69% of the time in Biology and Chemistry¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Raters with relevant curriculum background preferred the personalised item over the non-personalised item 0.68 of the time for Biology and 0.69 for Chemistry, significant by sign test against 0.5. → Kailai He 2026
Evidence¶
Kailai He 2026¶
Kailai He, Zhihao Wu, Linhai Zhang, Runcong Zhao, Yulan He, Jiazheng Li. (2026). CoLearn: An Agentic Tutor that Learns its Learner in a Human–AI Co-Learning Loop. https://arxiv.org/abs/2609.21154 doi:10.48550/arxiv.2609.21154
q2 · i? · design · r2
Blind A/B study with 18 raters, each completing 12 comparisons of an Adaptive versus a Random-topic question in randomised, unlabelled slots. The article reports "preferred 0.68 of the time across Biology raters" and 0.69 for Chemistry, with a two-sided sign test p= 0.008.
"The personalised item is preferred 0.68 of the time across Biology raters ( 0.74 and 0.73 for mixed- and strong-ability learners) and 0.69 across Chemistry raters ( p= 0.008 , sign test over 18 raters)."
Discussion¶
Related Claims¶
- In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related
- Students preferred AI-generated personalized videos over human-recorded non-personalized videos in direct comparison — related
- GenAI can provide personalised feedback in group work, but the depth of personalisation is contested — related
- LLM coders replicate many human discursive moves but lack questioning, adaptive responsiveness, and emotional expressiveness
- Automated metrics and LLM judges can misalign with expert human preferences in non-verifiable educational evaluation — related
- Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
collearn-ab-personalised-question-preference |
title |
In blind A/B comparison, raters prefer personalised (Adaptive) questions over non-personalised (Random-topic) questions about 68–69% of the time in Biology and Chemistry |
description |
In blind A/B comparison, raters prefer personalised (Adaptive) questions over non-personalised (Random-topic) questions about 68–69% of the time in Biology and Chemistry |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
source_checked |
2026-10-20 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — kailai-he-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.