Scrutable interfaces for generative AI: manipulable knowledge representations that let users shape outputs without understanding the model

Design · All designs
Evidence · 2 claims (2 for) · 1 study (1 design), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description

Scrutable interfaces use a knowledge representation, such as a concept map, to structure users' interactions with an underlying AI model. Critically, scrutable systems "don’t require users to understand how the underlying AI model works to change the AI system output"; instead the representation supports metacognitive self-monitoring, reflection, and planning. This article broadens the concept in two ways: extending scrutable systems from students to teachers as users, and applying them to black-box generative AI models, where the interface lets users "understand, reflect on, and directly edit the inputs to the GPT model to achieve their desired outcomes".

Design Implications

Context

Requirements

• A knowledge representation (e.g., a concept map) that users can create, edit, and iterate on, which feeds the AI's inputs
• An underlying AI model whose outputs change in response to the representation, so users need not understand the model's internals

Constraints

• Concept Catalyst does not allow teachers to scrutinize the inner workings of the GPT model; scrutability is limited to editing inputs
• Prior scrutable systems targeted students, not teachers, so this extension is exploratory

Target Learners

• K-12 engineering teachers

Learning Goals

• Reflective practice in curriculum and scaffold design
• Effective structuring of generative AI interactions

Claims

• Concept Catalyst Concept Map Supports Teacher Reflection [+M]
• Concept Catalyst Outputs Inspire Practice Reflection [+M]

Related Designs

• Four-model ITS architecture with inner and outer loop pedagogical model

Examples

-

Key Sources

• Gennie Mansi, Sunni Newton, Roxanne Moore, Meltem Alemdar, Mark Riedl. (2026). Concept Catalyst: Exploring Scrutable Interfaces to Structure K-12 Teacher Interactions with Generative AI. https://ceismc.gatech.edu

Page metadata

Field	Value
type	design
id	scrutable-interfaces-knowledge-representation-generative-ai
title	Scrutable interfaces for generative AI: manipulable knowledge representations that let users shape outputs without understanding the model
description	Scrutable interfaces use a knowledge representation, such as a concept map, to structure users' interactions with an underlying AI model.
status	review
generated	by process:settle-candidates on 2026-10-10
trust	unverified — no one has confirmed this page's content
sources	1 — gennie-mansi-2026

Frontmatter is the page's machine-readable record — see the Schema & Guide.