Human item-level scores on bash command exams are strongly bimodal, concentrated at zero and near-full credit with little intermediate partial credit¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Human evaluators most often awarded either no credit or near-full credit per item, with the 0% bin (n = 334) and 90% bin (n = 521) dominating and intermediate deciles low. → Alonso-Carracedo 2026

Evidence¶

Alonso-Carracedo 2026¶

Alonso-Carracedo, M., Fernandez-Boullon, R., Celard, P., Rodríguez-Martínez, F. J., Otero-Cerdeira, L. (2026). Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach. https://arxiv.org/abs/2607.02432 doi:10.48550/arxiv.2607.02432

q3 · i? · design · r2

Frequency distribution of normalized item-level scores assigned by the three human evaluators across the 1200 responses (Section 4.1.1, Figure 3). The article notes the 50% bin is the only notable intermediate concentration (n = 133), attributed to rubric provision for partial credit.

"The human distribution exhibits a strongly bimodal structure, with dominant peaks at the 0% bin (n= 334) and the 90% bin (n= 521), and a markedly lower frequency in the intermediate deciles (10%–80%)."

Discussion¶

Related Claims¶

• All four evaluated LLMs produce strongly bimodal item-level score distributions on bash exams, and rubric-enhanced prompts amplify the near-perfect-score peak — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• All four LLMs award a smaller share of available marks as bash question cognitive complexity increases, with L4 questions receiving the lowest proportions — related