Conduct field studies of GenAI-assisted coding under realistic development conditions, explicitly accounting for team structure, project complexity, and organizational processes.¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends that future productivity research place more emphasis on field studies conducted under realistic development conditions, in contrast to controlled laboratory settings. Professional software development involves large code bases with legacy code, coordination overhead, and strict quality and maintainability requirements. Such studies are needed to better assess the external validity of productivity effects and to explain the gap between laboratory and real-world findings observed in the meta-analysis, where laboratory effects (g = 0.73) substantially exceeded open-source (g = 0.01) and enterprise (g = 0.19) effects.

Design Implications¶

Context¶

Requirements¶

• Field studies must explicitly account for key confounding factors including team structure, project complexity, and organizational processes.
• Studies should examine interaction formats and implement GenAI tools in systematic comparisons across different GenAI technologies, including reasoning-capable GenAI, agentic systems, and emerging practices such as vibe coding.

Constraints¶

• Laboratory findings may not generalize to production environments where code bases are more complex and coordination costs are higher.
• Factors such as repository size, coordination overhead, legacy code, quality assurance processes, and specific interaction modes remain largely unobserved in available evidence.

Target Learners¶

• Professional software developers in enterprise and open-source settings

Target Learning Goals¶

• Understanding when and where GenAI tools improve real-world developer productivity

Related Strategies¶

• Sequence future adult-learning evaluation from usability observation to assistance-condition comparison with supervised independent assessment

Examples¶

-

Key Sources¶

• Maier, S., Gunzenhäuser, M., Schweisthal, J., Schneider, M., & Feuerriegel, S. (2026). A meta-analysis of the effect of generative AI on productivity and learning in programming. arXiv preprint. https://arxiv.org/abs/2605.04779. doi:10.48550/arxiv.2605.04779