Confusion-matrix annotation by a domain expert as a rapid parser-refinement methodology

Research Method · All research methods
Evidence · 1 claim (1 for) · 1 study (1 design), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description

A practical methodology for refining analytics tools with time-poor domain experts: the lecturer annotates AWA output pasted into Word using signal detection theory codes (True/False Positives/Negatives) in a 2×2 confusion matrix, with margin comments; the linguist responds in the shared document. The article describes this as "a rapid version of the more systematic coding that a qualitative data analyst would normally perform on a large corpus". The structured exchange builds a corpus from which precision, recall and F1 could in principle be calculated.

Accounts

• Confusion-matrix annotation by a domain expert as a rapid parser-refinement methodology: A practical methodology for refining analytics tools with time-poor domain experts: the lecturer annotates AWA output pasted into Word using signal detection theory codes (True/False Positives/Negatives) in a 2×2 confusion matrix, with margin comments; the linguist responds in the shared document. The article describes this as "a rapid version of the more systematic coding that a qualitative data analyst would normally perform on a large corpus". The structured exchange builds a corpus from which precision, recall and F1 could in principle be calculated. (Knight et al. (2016))

Claims

• Lexical adaptation of the general rhetorical parser reduced false positives in legal essay analysis without introducing parse errors [+W]

Related Research Methods

-

Key Sources

• Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (2016). Designing Academic Writing Analytics for Civil Law Student Self-Assessment. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-016-0121-0

Page metadata

Field	Value
type	research-method
id	confusion-matrix-annotation-by-a-domain-expert-as-a-rapid-parser-refinement-methodology
title	Confusion-matrix annotation by a domain expert as a rapid parser-refinement methodology
description	A practical methodology for refining analytics tools with time-poor domain experts: the lecturer annotates AWA output pasted into Word using signal detection theory codes (True/False Positives/Negatives) in a 2×2 confusion matrix, with margin comments; the linguist responds in the shared document.
status	review
generated	by process:settle-candidates on 2026-10-27
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.