Generative AI and Large Language Models for Software Engineering: Opportunities, Challenges, and Future Directions

Abstract

Artificial In this study, the impact of generative artificial intelligence (AI) and large language models (LLMs) on software engineering tasks was explored by applying a mixed methods experimental research design. The sample consisted of 64 software developers and computer science professionals, who were selected using purposive sampling to participate in the study, completing a series of programming, code generation, debugging, and documentation tasks in both an AI-assisted and non-AI-assisted environment. Data for task completion time, code accuracy, rate of defects and perceived productivity were analysed using paired-samples t-tests and semi-structured interviews with a subsample of 20 respondents were analysed qualitatively using thematic analysis to explore perceived benefits, limitations and concerns. The AI assistance was found to significantly speed up programming, code generation, and documentation tasks, further, there was a modest increase in perceived productivity for all tasks, except for debugging, which was not found to be significantly different. The accuracy of the debugging tasks did not show any statistically significant difference between the AI assistance and the human-only conditions. While logic defect rates were significantly lower when AI was used, the occurrence of security-relevant defects is significantly higher when using the AI-assisted code than the unassisted code. Thematic analysis of the interview data revealed three main benefits perceived: efficiency gains, learning support and reduced cognitive load, while concerns included hallucinated or inaccurate outputs, over-reliance and skill erosion, security review workload, and limited contextual understanding. Based on the findings of both quantitative and qualitative measures, this integration suggests that while generative AI and LLM-driven tools can enhance productivity and quality in specific, well-defined software engineering tasks, they also present new categories of risks and limitations, such as the complexity of debugging with these tools and the security concerns related to the output generated, that demand active human intervention and organizational measures to address.