Adding regex-based rules to few-shot LLM labeling improves agreement with human annotation (0.818 to 0.852 overall)¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Few-shot GPT-4o labeling evaluated against a 499-turn human-labeled gold set rose from 0.818 to 0.852 overall accuracy when regex-based rules were included in the prompts. → Menggian Wu 2026
q2 i? The largest dimension gain was Mastery-oriented aims (0.709 to 0.835, +0.126), while Verification-seeking showed a small decrease (0.834 to 0.829). → Menggian Wu 2026
Evidence¶
Menggian Wu 2026¶
Menggian Wu. (2026). Constructing Epistemic AI Literacy: Detecting Epistemic Aims and Processes in Student-AI Co-Programming. 2nd International Workshop on AI Literacy Education For All (ALIT4ALL 2026), co-located with AIED 2026, Seoul, Republic of Korea. https://arxiv.org/abs/2607.00211 doi:10.48550/arxiv.2607.00211
q2 · i? · design · r2
Evaluation against the 499-line human-annotated gold set compared two experimental conditions: few-shot prompts with versus without expert-derived regex principles. Overall accuracy improved 0.818 to 0.852, and 'Mastery-oriented_aims (0.709->0.835, +0.126)' showed the largest gain.
"Without regex-based rules, the LLM achieved an overall accuracy of 0.818 across seven binary dimensions. Adding lightweight regex-based rules improved overall accuracy to 0.852 (+0.035)"
Discussion¶
Related Claims¶
- 78.8% of student-GenAI turns lack mastery-oriented aims and reliable epistemic strategies — related
- GPT-4o's final-turn correctness labeling on CoMTA is only slightly less accurate than expert human annotators, which the authors read as close to human-level performance on a challenging task. — related
- Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. — related
- GPT-4o's correctness-labeling errors concentrate on final turns requiring numerical calculation, and its main KC-labeling error is assigning too few standards to a turn. — related
- In-context learning did not consistently improve LLM annotation reliability over zero-shot prompting for engagement coding — related
- A pilot human annotation surfaces a framing discrepancy: a NEET aspirant labeled all 10 sampled questions as Recall, including the five the LLM tagged as Elimination/Negation — related
- In classroom video labeling, high raw agreement (often exceeding 90%) co-occurred with low chance-corrected κ due to temporal misalignment — related
- Only 11.1% of interactions pair mastery-oriented aims with advanced strategies such as epistemic justification — related
- The framework generalizes under automatically generated Q-matrices without requiring expert annotation — related
- The FastFit classifier slightly outperforms GPT-4 used directly under the same closed-label prompt on the same gold set (0.800 vs. 0.786 accuracy) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
regex-rules-improve-fewshot-llm-labeling-accuracy |
title |
Adding regex-based rules to few-shot LLM labeling improves agreement with human annotation (0.818 to 0.852 overall) |
description |
Adding regex-based rules to few-shot LLM labeling improves agreement with human annotation (0.818 to 0.852 overall) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — menggian-wu-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.