All four evaluated LLMs produce strongly bimodal item-level score distributions on bash exams, and rubric-enhanced prompts amplify the near-perfect-score peak¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? GPT 5.2, Claude Opus 4.6, Gemini 3.0 Pro and GLM 5 all show a dominant 90% peak and secondary 0% peak, with Variant 2 generally amplifying the 90% peak while modestly reducing the 0% bin. → Alonso-Carracedo 2026

Evidence¶

Alonso-Carracedo 2026¶

Alonso-Carracedo, M., Fernandez-Boullon, R., Celard, P., Rodríguez-Martínez, F. J., Otero-Cerdeira, L. (2026). Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach. https://arxiv.org/abs/2607.02432 doi:10.48550/arxiv.2607.02432

q2 · i? · causal · r2

Descriptive distribution analysis of the four LLMs' grades under both prompt variants on the same 1200 responses (Section 4.2.1, Figure 5). The article states "The V2 variants generally amplify the 90% peak while modestly reducing the 0% bin" and that Gemini is most extreme in bimodality.

"All four models share a common structural feature: a strongly right-skewed distribution with a dominant peak at 90% and a secondary at 0%, indicating that item-level scoring is inherently bimodal in nature."

Discussion¶

Related Claims¶

• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Human item-level scores on bash command exams are strongly bimodal, concentrated at zero and near-full credit with little intermediate partial credit — related
• Rubric-enhanced prompting (Variant 2) raises LLM assigned scores relative to the no-rubric baseline, with model-specific gaps at particular taxonomy levels — related
• All four LLMs award a smaller share of available marks as bash question cognitive complexity increases, with L4 questions receiving the lowest proportions — related
• Despite generally accurate feedback, the LLM's scoring explanations could be unpredictable and prone to logical inconsistency, failing to award points even when citing the correct rubric directive — related
• Frontier LLMs produced near-perfect solutions to CS1 labs, with mean hidden-instructor-test scores of 98.93–99.99% across model–year combinations — related
• Average H-scores rose from near-zero in pre-LLM offerings to 11.9 in Spring 2026, with maximum scores reaching 81 — related