Multilingual ASAP content scoring datasets in five languages

Product or Programme · All products and programmes
Evidence · 2 claims (2 for) · 1 study (1 design), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description

A publicly released collection of Chinese, English, French, German, and Spanish science short-answer responses with native-speaker annotations for research on automated content scoring, maintained by Andrea Horbach and collaborators.

Components

• Multilingual ASAP content scoring datasets in five languages with three newly collected variants, publicly released: The article presents language variants of the ASAP short-answer content scoring dataset covering Chinese, English, French, German and Spanish for three science-related prompts. Three variants were newly collected via crowdsourcing (ASAPen, ASAPes, ASAPfr, about 300 usable answers per prompt), alongside ASAPde and a Chinese dataset from high school students. Each was double-annotated by native speakers using the original scoring guidelines, with inter-annotator agreement in QWK reported per prompt (e.g. .83-.69 for ASAPen). The article states the data "is publicly available under https://github.com/andreahorbach/CrosslingualScoring/". (Andrea Horbach et al. (2024))

Claims

• Under unseen-answers evaluation, the German Transformer model's F-Scores are numerically similar to state-of-the-art Transformer results for English [+W]
• The German Transformer model's tokenization handles native-speaker morphology well but struggles with learner-language errors, possibly explaining lower CREG results [+W]

Related Products and Programmes

-

Key Sources

• Andrea Horbach, Joey Pehlke, Ronja Laarmann-Quante, Yuning Ding. (2024). Crosslingual Content Scoring in Five Languages Using Machine-Translation and Multilingual Transformer Models. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-023-00370-1

Page metadata

Field	Value
type	product
id	multilingual-asap-content-scoring-datasets-in-five-languages
title	Multilingual ASAP content scoring datasets in five languages
description	A publicly released collection of Chinese, English, French, German, and Spanish science short-answer responses with native-speaker annotations for research on automated content scoring, maintained by Andrea Horbach and collaborators.
status	review
product_kind	dataset
generated	by process:settle-candidates on 2026-10-27
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.