Cohort members predominantly used off-the-shelf multimodal LLMs, mostly OpenAI's GPT, and nearly all combined closed and open source models¶

Claim · All claims
Evidence · 1 study · qualitative r? · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? All but two partners used a combination of closed and open source models, and most partners used OpenAI's GPT across versions 3.0, 3.5, 4.0, and 4.0 Turbo. → models-used

Evidence¶

models-used¶

Noakes, S., Shell, A., Murillo, A., Van Nostrand, P., Ruiz, P., Cornick, S., and Karim, S. (2024, October). An Ethical and Equitable Vision of AI in Education: Learning Across 28 Exploratory Projects. Digital Promise. https://doi.org/10.51388/20.500.12265/232

q2 · i? · qualitative · r?

Findings on multimodal large language models leveraged by cohort members, reported in the Technical Findings section. The report states there was a general preference for off-the-shelf models as easier to access and use, though they tended to be more black box with less insight into training.

"All but two partners described using a combination of closed and open source models."

Discussion¶

Related Claims¶

• Open-weight (DeepSeek) and proprietary (OpenAI) rubrics produced equivalent grading outcomes in every assignment-approach comparison except Assignment 1 free-form
• In automated evaluation, equivalence with the human baseline depended on the evaluator model: DeepSeek V3 met equivalence for all rubric sources while GPT-4.1 and GPT-4o scored below the baseline
• The OpenAI gpt-4o reference model outperformed all quantised models on human-rated coherence, fluency, relevance and cultural appropriateness, serving as an upper benchmark
• Closed-source LLMs reach CSE Level 6 while most open-source baselines cluster at CSE Levels 3–4 on CSEBench — related