The workflow generalizes beyond propositional logic, explaining 79.39% of modal logic pairs and 35.89% of CTL pairs¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On the modal logic dataset (12482 pairs) the workflow's transformations explain 79.39% of pairs, comparable to the propositional case. → Lilian Killich 2026 q2 i? On the CTL dataset (7210 pairs) the workflow's transformations explain 35.89% of pairs, with the two largest transformations explaining 2.98% and 2.48%. → Lilian Killich 2026 (2)

Evidence¶

Lilian Killich 2026¶

Lilian Killich, Marko Schmellenkamp, Fabian Vehlken, Thomas Zeume. (2026). Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs. https://arxiv.org/abs/2609.17111 doi:10.48550/arxiv.2609.17111

q2 · i? · design · r2

Second experiment set applying the workflow to a modal logic dataset of 12482 pairs (3951 distinct) from prior work; the article reports coverage of "79.39% of the pairs in the data set". No effect size printed.

"For modal logic, the transformations found by our workflow explain 79.39% of the pairs in the data set, which is comparable to the propositional logic case."

Lilian Killich 2026 (2)¶

Lilian Killich, Marko Schmellenkamp, Fabian Vehlken, Thomas Zeume. (2026). Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs. https://arxiv.org/abs/2609.17111

q2 · i? · design · r2

Application to a CTL dataset of 7210 pairs (2695 distinct) collected by the authors with an interactive learning tool in two iterations of an introductory logic course; coverage is "35.89% of the pairs", lower than the other formalisms.

"For CTL, the transformations found by our workflow explain 35.89% of the pairs with the two transformations that explain the largest fractions of the data set being"

Discussion¶

Related Claims¶

• Educator inspection found most large clusters' main transformations suitable for feedback, though many required sub-transformations for targeted feedback — related
• On the full propositional logic dataset, LLM-found bug fixing transformations explain 84.44% of pairs versus 71.57% for hand-picked mistakes — related
• The workflow discovers new mistake candidates not explained by prior hand-picked mistakes, such as an exclusive-or-instead-of-negated-conjunction mistake — related
• Across linguistic-operator subdatasets, workflow-found transformations explain at least as many pairs as hand-picked mistakes in most cases — related