Triangulate LLM-based discourse measures with multiple methods and seek input from participating communities¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends that scholars using LLM-based discourse measures "triangulate data from multiple methods and seek input from participating communities to validate and improve measures." This participatory validation strategy treats the people being measured as participants in deciding what the data means, grounding measures in youth lived experience. It is framed as especially important for combating biases and harms LLMs can reproduce that disproportionately affect marginalized youth.

Design Implications¶

Context¶

Requirements¶

• Relationship-building with students over time to enable member checks and retrospection
• Access to the communities whose talk is being measured

Constraints¶

• The article notes the same methods that allow re-contextualizing a small number of conversations constrain how many can be analyzed

Target Learners¶

• K-12 students, particularly multilingual and marginalized youth

Target Learning Goals¶

• accurate and equitable measurement of student discourse and participation

Affordances¶

• Epistemic Shift Youth Validation Llm Talk Measures

Related Strategies¶

• Use LLM persona simulation only for preliminary experimentation, stress testing, and hypothesis generation, with separate validation for consequential applications
• Engage community constituents through multiple input methods as the first step in POG development
• Involve key stakeholders directly in defining privacy and designing LA systems

Examples¶

-

Key Sources¶

• Liliana Santos-Deonizio, James Malamut, Ramón Antonio Martínez, and Dorottya Demszky. (2027). When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk. In Proceedings of TBD (CSCW '27). ACM, New York, NY, USA. https://arxiv.org/abs/2608.23780