On the DisagreementSet, data augmentation improved the multi-label classifiers' exact match ratio and F1 score, while both ratios declined relative to the AgreementSet test¶
Claim · All claims
Evidence · 1 study · designr2·q3
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Testing the best-performing classifiers on the DisagreementSet, using the data augmentation method improved the exact match ratio and the F1 score compared with the classifiers without DA. → Yuanyuan Hu 2023
Evidence¶
Yuanyuan Hu 2023¶
Yuanyuan Hu. (2023). Can Multi-Label Classifiers Help Identify Subjectivity? A Deep Learning Approach to Classifying Cognitive Presence in MOOCs. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-022-00310-5
q3 · i? · design · r2
Validation test of the multi-label classifiers on the coder-disagreement messages (Table 6). "The exact match ratio and the F1 score were improved by using the DA method", while relative to the AgreementSet test the exact match and fully incorrect ratios largely declined.
"The exact match ratio and the F1 score were improved by using the DA method. Compared to the results of the AgreementSet in Table 4, we found the F1 score reached the similar value in the test with the DA method, but the exact match ratio and the fully incorrect ratio largely declined."
Discussion¶
Related Claims¶
- Multi-label BERT classifiers slightly outperformed single-label Random Forest baselines on example-based F1 scores but under-performed with lower exact match ratios on MOOC cognitive presence classification — related
- Tested on coder-disagreement messages, multi-label classifiers labelled most messages into two adjacent phases with one predicted label matching the manual label, and partially correct predictions exceeded half of the set — related
- Multi-label classifiers predicted MOOC messages as one label or two adjacent cognitive presence phases only, while expert coders produced non-adjacent and three-phase labels on almost 4% of messages — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
data-augmentation-improves-disagreementset-performance |
title |
On the DisagreementSet, data augmentation improved the multi-label classifiers' exact match ratio and F1 score, while both ratios declined relative to the AgreementSet test |
description |
On the DisagreementSet, data augmentation improved the multi-label classifiers' exact match ratio and F1 score, while both ratios declined relative to the AgreementSet test |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2023 |
generated |
by process:wiki-ingest on 2026-10-27 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — yuanyuan-hu-2023 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.