Hint-based prompts raise ChatGPT scoring agreement with human experts to 98–100%, while no-hint prompts are unstable (Q2 precision 0.42)¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Under the hint condition, agreement of 98% or higher was achieved for all questions, with perfect agreement (100%) for Q1 and Q2; under the no-hint condition results varied, with Q2 precision only 0.42. → Satoshi Takahashi 2026 q3 i? When ChatGPT scored its own generated problem, perfect agreement was observed (Fleiss' κ = 1.00 for Q1 and Q2). → Satoshi Takahashi 2026 (2)

Evidence¶

Satoshi Takahashi 2026¶

Satoshi Takahashi, Atsushi Yoshikawa, Megumi Kose, Kenichi Suzuki, Chieko Inoue, Yumi Watanabe, and Mari Sawada. (2026). Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence. arXiv. https://arxiv.org/abs/2609.25790 doi:10.48550/arxiv.2609.25790

q3 · i? · causal · r2

Experiment 2 compared two prompt conditions (no-hints vs. hints with human-scored examples) on descriptive answers to the ChatGPT-generated problem; scoring was repeated nine times with majority vote and evaluated with accuracy, precision, recall, and F1.

"By contrast, under the hint condition, an agreement of 98% or higher was achieved for all questions, with perfect agreement (100%) recorded for Q1 and Q2."

Satoshi Takahashi 2026 (2)¶

Satoshi Takahashi, Atsushi Yoshikawa, Megumi Kose, Kenichi Suzuki, Chieko Inoue, Yumi Watanabe, and Mari Sawada. (2026). Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence. arXiv. https://arxiv.org/abs/2609.25790

q3 · i? · design · r2

Experiment 1 agreement analysis (Table 4): 9-ChatGPT agreement on the ChatGPT-generated problem reached 100.00% for Q1 and Q2 and 97.67% for Q3, with Fleiss' κ of 1.00, 1.00, and 0.97.

"Furthermore, when ChatGPT itself scored the ChatGPT-generated problem, perfect agreement was observed, confirming even higher consistency."

Discussion¶

Related Claims¶

• ChatGPT zero-shot relevance extraction for instruction-quality assessment retrieves mostly irrelevant utterances and overestimates instruction quality — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• ChatGPT's exam scores were within 10% of human grades 70% of the time in a study of AI-based grading — related
• AI scoring consistency approaches or exceeds 90% for most rubric items, but spelling is replaced by a deterministic LanguageTool-based approach achieving 68% accuracy with 100% consistency — related
• ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related
• LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes — related
• Switching from reference-based to contextual evaluation prompts improved GenAI grading accuracy — related
• Agentic assessment systems report grading time reductions of 50–70% with 90–95% agreement with human graders — related
• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — a broader claim this one bears on
• Two of seven frontier LLMs show behavioral inconsistency between monitored and unmonitored conditions (Grok-4.1-fast 5.50; Gemini-3-Pro 5.00), while four show perfect consistency (1.0) — related
• Automated HDR problem generation makes repeated and longitudinal assessment feasible by producing structurally equivalent cases with different contexts — related