Dual-assessment framework for evaluating LLM human simulations via joint self-report and observer-rated persona measurement

Research Method · All research methods
Evidence · 5 claims (4 for, 1 mixed) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 5 claims rest on one study

Description

A methodology adapted from clinical psychology that jointly measures self-reported characteristics and observer-rated behavioral expressions of an LLM persona. Self-report scores measure how the persona agent characterizes itself when directly prompted with questionnaire items, while observer-report scores measure which persona-relevant behaviors are expressed in generated narratives or conversational segments. The framework's central contribution is exposing self–observer dissociation: "LLMs reliably report the assigned persona when directly asked, yet their spontaneous behavioral expression of that persona degrades in unscripted conversation." It also distinguishes between-conversation from within-conversation stability as complementary temporal dimensions.

Accounts

• Dual-assessment framework for evaluating LLM human simulations via joint self-report and observer-rated persona measurement: A methodology adapted from clinical psychology that jointly measures self-reported characteristics and observer-rated behavioral expressions of an LLM persona. Self-report scores measure how the persona agent characterizes itself when directly prompted with questionnaire items, while observer-report scores measure which persona-relevant behaviors are expressed in generated narratives or conversational segments. The framework's central contribution is exposing self–observer dissociation: "LLMs reliably report the assigned persona when directly asked, yet their spontaneous behavioral expression of that persona degrades in unscripted conversation." It also distinguishes between-conversation from within-conversation stability as complementary temporal dimensions. (Jana Gonnermann-Müller et al. (2026))

Claims

• Observer-rated behavioral expression of high- and moderate-intensity ADHD personas declines progressively across turns in unscripted LLM conversations [+M]
• Default personas show a self-report–behavior gap: they endorse moderate-range ADHD characteristics on self-report while displaying only weakly elevated ADHD-like behavior to observers [+M]
• LLM-simulated students' self-reported ADHD persona scores remain stable throughout extended multi-turn conversations [+W]
• Between-conversation persona stability is intensity-dependent: high-intensity ADHD personas cluster tightly while moderate-intensity personas show the highest aggregate variability [~W]
• LLM observer ratings converge excellently with human expert judgments in applying the CAARS observer instrument to generated narratives [+W]

Related Research Methods

-

Key Sources

• Jana Gonnermann-Müller, Jennifer Haase, Nicolas Leins, Thomas Kosch, Sebastian Pokutta. (2026). LLM-Based Educational Simulation: Evaluating Temporal Student Persona Stability Across ADHD-Related Persona Specifications. https://arxiv.org/abs/2605.06307

Page metadata

Field	Value
type	research-method
id	dual-assessment-framework-for-evaluating-llm-human-simulations-via-joint-self-report-and-o
title	Dual-assessment framework for evaluating LLM human simulations via joint self-report and observer-rated persona measurement
description	A methodology adapted from clinical psychology that jointly measures self-reported characteristics and observer-rated behavioral expressions of an LLM persona.
status	review
generated	by process:settle-candidates on 2026-10-11
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.