Email-detection confidence judgments show good calibration but poor resolution, with aggregate overconfidence of 18%¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2022: tool and model accuracy changes quickly

Subclaims¶

q3 i? Across two experiments, participants' mean confidence (85%) exceeded mean accuracy (67%), and calibration was generally good while resolution was very poor. → Casey Inez Canfield & Baruch Fischhoff & Alex Davis 2019

Evidence¶

Casey Inez Canfield & Baruch Fischhoff & Alex Davis 2019¶

Casey Inez Canfield & Baruch Fischhoff & Alex Davis. (2019). Better beware: comparing metacognition for phishing and legitimate emails. Metacognition and Learning. https://doi.org/10.1007/s11409-019-09197-5

q3 · i? · causal · r2

Pooled analysis of two experiments (152 mTurk and 98 SBO participants; 250 total) scoring detection judgments with the Brier decomposition. The article reports "aggregate overconfidence of 18%" and states calibration was generally good while "resolution was very poor."

"Across all questions and individuals, mean confidence was 85% ( SD =8 % ) . Mean accuracy was much lower at 67% ( SD = 11%), making for aggregate overconfidence of 18%"

Discussion¶

Related Claims¶

• Calibration is worse and resolution better for phishing emails than for legitimate emails — related
• Future anti-phishing interventions should measure both performance and metacognition, aiming to shift the phishing calibration curve toward the legitimate-email pattern — related
• Participants are consistently overconfident for phishing emails but relatively well calibrated for legitimate emails except at 100% confidence — related
• Phishing email difficulty varies widely (15–85% correct) while mean confidence varies little (80–90%), leaving some victims unaware they were fooled — a narrower finding that bears on this claim
• Individual and task factors fail to predict resolution, while perceived consequences predict calibration in opposite directions for phishing and legitimate emails — related
• Better resolution on the laboratory phishing task predicts lower odds of malicious files on participants' home computers — related
• Most preservice teachers were overconfident in the completeness of their fraction comparison explanations at both first draft and revision — a broader claim this one bears on
• A theoretical article, citing two simulation studies, reports that novice clinical learners overestimated their performance and that their confidence exceeded their accuracy — a broader claim this one bears on