LLMs perform worse on complex, open-ended, or subjective assignments than on short, clearly formulated answers¶

Claim · All claims
Evidence · 1 study (2 entries) · review r3 · q2–q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? AI effectiveness often decreases for complex assignments requiring broader context or subjective judgment, while it performs very well with short, clearly formulated answers. → Marcin Jukiewicz and Michał Wyrwa 2026 q2 i? In one reviewed chemistry study, GPT-4's F1 for long, multi-faceted factual answers was 0.69 versus 0.74 for short responses. → Marcin Jukiewicz and Michał Wyrwa 2026 (2)

Evidence¶

Marcin Jukiewicz and Michał Wyrwa 2026¶

Marcin Jukiewicz and Michał Wyrwa. (2026). Can ChatGPT Replace the Teacher in Assessment? A Review of Research on the Use of Large Language Models in Grading and Providing Feedback. Appl. Sci. 2026, 16, 680. https://doi.org/10.3390/app16020680

q3 · i? · review · r3

The review's RQ4 synthesis of task-type moderators across the included studies states that AI "performs very well with short, clearly formulated answers" but declines on complex assignments. No effect size is printed.

"Artificial intelligence performs very well with short, clearly formulated answers; however, its effectiveness often decreases when it comes to more complex assignments requiring broader context or subjective judgment"

Marcin Jukiewicz and Michał Wyrwa 2026 (2)¶

Marcin Jukiewicz and Michał Wyrwa. (2026). Can ChatGPT Replace the Teacher in Assessment? A Review of Research on the Use of Large Language Models in Grading and Providing Feedback. Appl. Sci. 2026, 16, 680. https://doi.org/10.3390/app16020680

q2 · i? · review · r3

The review reports, from one university chemistry study in its evidence table, that GPT-4 scored F1 = 0.69 on long multi-faceted answers versus 0.74 on short responses. Only test-level metrics are printed; no standardized effect size.

"Language models (LLMs, e.g., GPT-4) perform worse in grading long, multi-faceted, factual answers than in grading short responses—even with advanced techniques and rubrics, F1 for GPT-4 was only 0.69 (for short responses, 0.74)."

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — related
• Falsifiable hypothesis: unrestricted AI use on deep-processing tasks will produce a product–process dissociation—higher-rated assignments but lower unaided delayed transfer — related
• Reviewed literature reports risks of overreliance, AI errors in complex tasks, and students' uncritical acceptance of AI-generated outputs — related
• LLM performance drops from offline to real-time proxy evaluation when clues arrive incrementally — related
• Students view AI videos as appropriate for simple, visualizable, and supplemental learning contexts but inappropriate for complex, subjective, or interactive ones — related
• Heavy reliance on generative AI can undermine learning: students with GPT-4 access performed better on practice problems but worse on exams once the tool was removed — related
• LLM-generated feedback is more comprehensive in volume than human feedback but is sometimes too generic, abstract, or misaligned with the assigned grade — a narrower finding that bears on this claim
• ChatGPT shows high success on open-ended general chemistry exam questions but lower accuracy at the university level, and unverified copying of its answers is a reported risk — related
• Task complexity moderates whether users experience the constructive information search process — related