All four LLMs award a smaller share of available marks as bash question cognitive complexity increases, with L4 questions receiving the lowest proportions¶
Claim · All claims
Evidence · 1 study · causalr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Across all models and variants, assigned score proportions decrease monotonically from L1 to L4, mirroring the human pattern of declining performance with cognitive complexity. → Alonso-Carracedo 2026
Evidence¶
Alonso-Carracedo 2026¶
Alonso-Carracedo, M., Fernandez-Boullon, R., Celard, P., Rodríguez-Martínez, F. J., Otero-Cerdeira, L. (2026). Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach. https://arxiv.org/abs/2607.02432 doi:10.48550/arxiv.2607.02432
q2 · i? · causal · r2
Per-taxonomy-level analysis of LLM assigned grades under both variants (Section 4.2.2, Figure 6). The article reports L1 questions receive the highest proportions and L4 the lowest for every model, with GPT V2 (50.80%) and Gemini V2 (50.64%) highest at L4.
"A consistent pattern emerges across all four models: the proportion of the available mark assignedtoquestionsdecreasesmonotonicallyascognitivecomplexityincreases."
Discussion¶
Related Claims¶
- All four evaluated LLMs produce strongly bimodal item-level score distributions on bash exams, and rubric-enhanced prompts amplify the near-perfect-score peak — related
- Rubric-enhanced prompting (Variant 2) raises LLM assigned scores relative to the no-rubric baseline, with model-specific gaps at particular taxonomy levels — related
- Human item-level scores on bash command exams are strongly bimodal, concentrated at zero and near-full credit with little intermediate partial credit — related
- Taxonomy level is defined as the maximum of cognitive complexity and operational impact, ensuring monotone coverage of both dimensions — related
- Proportions of variance explained follow a curvilinear pattern across the six taxonomy levels — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-scores-decline-with-cogtax-level |
title |
All four LLMs award a smaller share of available marks as bash question cognitive complexity increases, with L4 questions receiving the lowest proportions |
description |
All four LLMs award a smaller share of available marks as bash question cognitive complexity increases, with L4 questions receiving the lowest proportions |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — alonso-carracedo-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.