The fine-tuned BERT model generalizes to an external dataset collected by a different research team, maintaining almost perfect agreement with human coders¶
Claim · All claims
Evidence · 1 study · designr2·q3
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because a design page cites it; treat it as a lead to check, not an established finding.
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? On an external dataset of 7th-grade essays from another research team, the model achieved Cohen's Kappa of 0.86 with human coders (n = 150). → Ye 2026
Evidence¶
Ye 2026¶
Ye, C., Borman, T. H., & Borman, G. D. (2026). Automating Self-Affirmation Essay Coding: Fine-Tuned BERT Performance Comparable to Human Coders and Comparison with GPT-4. Journal of Educational Data Mining, Volume 18, No 1. https://github.com/visortown/bert-self-affirm
q3 · i? · design · r2
Generalization test applying the fine-tuned model to an external dataset of 4,660 students in 25 schools collected by another organization in 2023-2024, with reliability coding on a stratified random subsample of 150 essays yielding "a Cohen’s Kappa of 0.86".
"The model achieved a Cohen’s Kappa of 0.86 with human coders (n = 150) , indicating almost perfect agreement (McHugh, 2012)."
Discussion¶
Related Claims¶
- The fine-tuned BERT model's classification performance is consistent across stereotype-threatened and non-threatened students and other demographic subgroups — related
- A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
- NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — a broader claim this one bears on
- Fine-tuned BERT outperforms a Naive Bayes baseline and zero-shot GPT-4 in classifying self-affirmation essays — related
- Transformer-based NLP models achieve automated essay scoring with correlations to human grading exceeding 0.85 — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
bert-generalizes-external-self-affirmation-dataset |
title |
The fine-tuned BERT model generalizes to an external dataset collected by a different research team, maintaining almost perfect agreement with human coders |
description |
The fine-tuned BERT model generalizes to an external dataset collected by a different research team, maintaining almost perfect agreement with human coders |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
source_checked |
2026-10-20 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-09 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — ye-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.