Across rubric dimensions, content scored highest (A = 4.03) while production scored lowest (D = 3.41), driven by synthetic voice quality (D1 = 3.16)¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Content was the strongest rubric dimension (A = 4.03, SD = 0.67; 99% of videos at or above 3), and production the weakest (D = 3.41), driven by spoken-language quality D1 = 3.16 (SD = 1.09). → Romain Puech 2026

Evidence¶

Romain Puech 2026¶

Romain Puech, Dewang Kumar Agarwal, Antonio Santamaría Escobar, Dimitris Bertsimas. (2026). Bespoke: Generating MOOC-Quality Industry-Personalized Lecture Videos at Scale. arXiv preprint. https://arxiv.org/abs/2609.26540 doi:10.48550/arxiv.2609.26540

q2 · i? · design · r2

Descriptive results from the same 92-video expert rating study, reporting mean scores per rubric dimension. The article states "Content is the strongest dimension" at A = 4.03 and "Production is weakest" at D = 3.41, driven by voice (D1 = 3.16, SD = 1.09).

"Content is the strongest dimension (𝐴= 4.03, SD = 0.67): 99% score≥ 3and 82% score≥ 4. Personalization depth𝐵1 = 3.86(94.6% ≥ 3); audience calibration 𝐵2 = 3.52. Pedagogy𝐶= 3.57, led by narrative coherence (𝐶1=3.79). Production is weakest (𝐷=3.41), driven by voice (𝐷1=3.16, SD=1.09)."

Discussion¶

Related Claims¶

• Domain-matched experts judged 87% of Bespoke-generated lecture videos at or above the rubric midpoint corresponding to a standard MOOC lecture's quality (mean G = 3.42 of 5) — related
• Pedagogy and production dimensions associated most strongly with global quality (Spearman ρ = 0.68 and 0.66), while content was relatively independent (ρ = 0.46) — related
• Reviewer free-text comments identified synthetic voice, slide–narration reveal timing, and slide layout as the main remaining quality issues — related
• Industry-targeted videos scored only slightly higher on personalization depth than generic-audience videos (0.32 points; modeled increment 0.25, p = 0.11), an effect not resolved under reviewer clustering — related
• LLM scoring performance is dimension-specific: Reasoning shows the strongest agreement and Terminology the weakest across all three strategies — related
• Multimodal XGBoost models achieved rubric-aligned scoring comparable to expert ratings, and no single modality approached multimodal performance — related
• Activity-Technology alignment is the weakest quality dimension across all agents (mean = 2.19), identifying technology integration as a universal challenge — related
• Among 6,699 LLM-judge deduction flags, visual delivery is the largest deficiency category (39%) and learner adaptation the second (27%), while content problems are rarer (16%) — related