Skip to content

LLM-queries

Product or Programme · All products and programmes
Evidence · 1 claim (1 mixed) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description

A GitHub repository released by Keith et al. containing the code for prompting multiple large language models, processing student essay text, merging model and human ratings, and computing agreement and bias measures.

Components

  • LLM-queries GitHub repository for LLM essay-coding analysis pipeline: The authors released the code for their LLM prompting and analysis pipeline, which processed anonymized student paper PDFs into text files, applied each prompt to each LLM via OpenRouter.ai with unified API-level control and temperature 0, merged LLM ratings with human ratings, and computed correlation, absolute error, and normalized signed bias measures. The article states: "The code is available at https://github.com/imrryr/LLM-queries". It supports replication of the 60,060-prompt study design. (Keith et al. (2026))

Claims

Key Sources

  • Keith, Shelley; Pavlik, Philip I., Jr.; Stives, Kristen L.; Kerr, Laura Jean. (2026). Comparing Zero-Shot Large Language Model Prompting with Human Coding of Theory Concepts in Student Essays. Journal of Educational Data Mining, Volume 18, No 1. https://github.com/imrryr/LLM-queries
Page metadata
Field Value
type product
id llm-queries
title LLM-queries
description A GitHub repository released by Keith et al.
status review
moved_from element/llm-queries-github-analysis-code
product_kind software
generated by process:sweep-kinds on 2026-10-10
trust unverified — no one has confirmed this page's content
sources 1 — keith-2026

Frontmatter is the page's machine-readable record — see the Schema & Guide.