A FACTRIA-aware chatbot increases the number of biasing sub-factors stakeholders consider when interpreting institutional analytics visualizations¶

Claim · All claims
Evidence · 1 study (3 entries) · qualitative r2–r3 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 Across all collected data (n = 22 case observations), using the FACTRIA-aware chatbot significantly increased the average number of sub-factors addressed per case, with a very large effect size (d = 2.1530). → Marques 2026 q2 i3 The increase holds when restricting to participants' first assigned case (n = 11, d = 4.4947) and second assigned case (n = 11, d = 1.7866), both statistically significant. → Marques 2026 (2) q2 i? Before chatbot interaction, participants referenced an average of only 1.50 sub-factors (SD = 0.80) despite having read the framework, indicating familiarity alone was insufficient. → Marques 2026 (3)

Evidence¶

Marques 2026¶

Marques, F., Ortiz-Beltrán, A., Amarasinghe, I., & Hernández-Leo, D. (2026). Responsible Institutional Analytics: Interpreting Bias with AI Support. Journal of Universal Computer Science, 33(6). https://arxiv.org/abs/2610.07205 doi:10.48550/arxiv.2610.07205

q2 · i3 · causal · r1

Quantitative analysis of the think-aloud study treating each case addressed by a participant as an independent observation (n = 22). The paired-samples t-test showed "a statistically significant difference between without and with the F ACTRIA-aware chatbot, p < . 001" with Cohen's d = 2.1530.

"We computed the paired-samples t-test, which showed a statistically significant difference between without and with the F ACTRIA-aware chatbot, p < . 001. and Cohen￿sd = 2 .1530, indicating a very large effect size."

Marques 2026 (2)¶

Marques, F., Ortiz-Beltrán, A., Amarasinghe, I., & Hernández-Leo, D. (2026). Responsible Institutional Analytics: Interpreting Bias with AI Support. Journal of Universal Computer Science, 33(6). https://arxiv.org/abs/2610.07205. doi:10.48550/arxiv.2610.07205

q2 · i3 · qualitative · r3

First-case-only scenario (n = 11), computed to control potential learning effects from completing a second case. The article reports "statistically significant differences, with p < . 001 and Cohen￿sd = 4.4947". A third scenario (second cases only, n = 11) also showed significant differences with d = 1.7866.

"In this second scenario, we also found statistically significant differences, with p < . 001 and Cohen￿sd = 4.4947, indicating a very large effect size."

Marques 2026 (3)¶

Marques, F., Ortiz-Beltrán, A., Amarasinghe, I., & Hernández-Leo, D. (2026). Responsible Institutional Analytics: Interpreting Bias with AI Support. Journal of Universal Computer Science, 33(6). https://arxiv.org/abs/2610.07205. doi:10.48550/arxiv.2610.07205

q2 · i? · qualitative · r2

Descriptive baseline from the discussion of the think-aloud study: without chatbot support participants averaged "1.50 sub-factors ( SD = 0.80)" across both cases. The authors note all participants had read the framework beforehand, so familiarity alone did not prompt broad consideration.

"Before interacting with the F ACTRIA-aware chatbot, participants referenced an average of 1.50 sub-factors ( SD = 0.80) considering both cases, even though each case required considering multiple dimensions, from data quality and aggregation to course design and demographics."

Discussion¶

Related Claims¶

• Qualitative case narratives show the chatbot shifted stakeholders from narrow, visually driven readings toward conditional, context-aware interpretations — possibly the same claim (merge candidate)
• The FACTRIA-aware chatbot received good usability ratings (SUS 70.84) and high usefulness ratings from most of the 11 stakeholders — related
• Transition network analysis shows the chatbot's prompts and explanations structured the interaction while participants generated their own interpretations — related
• Stakeholder survey rated achievability lowest across Framework elements, especially Element 5 (continuous improvement) — related