Development of an Agile Story Point Estimation Model for Scrum: A Fusion of Natural Language Processing and Machine Learning Techniques

Abstract

Accurate Software Effort Estimation (SEE) is a persistent challenge in Agile Software Development (ASD) due to the iterative and dynamic nature of Agile processes, where traditional estimation techniques like Planning Poker and T-shirt Sizing are susceptible to biases and often lead to cost overruns and schedule delays. Addressing these limitations, this doctoral thesis introduces a novel automated SEE model that integrates advanced Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL) methodologies to improve the precision and reliability of Story Point (SP) estimations. The model employs the Sentence Bidirectional Encoder Representations from Transformers (SBERT) architecture for extracting rich semantic features from textual descriptions of user stories (USs) and issues, paired with Gradient Boosted Tree (GBT) algorithms, including Category Boosting (CatBoost), eXtreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LightGBM), to predict development efforts accurately.

The model underwent rigorous training and validation on a comprehensive dataset comprising 31,960 issues from 26 open-source Agile projects. This evaluation was structured around three key research questions (RQs) to assess predictive accuracy, reliability, and generalizability. Findings reveal that the proposed model significantly outperforms baseline estimators, including Random Guessing, Mean, and Median Estimators, as well advanced as state-of-the-art models like Deep Software Estimation (Deep-SE), Term Frequency-Inverse Document Frequency Software Estimation (TF-IDF-SE), Latent Dirichlet Allocation (LDA)-based Hierarchical Clustering for SP Estimation (LHC-SE), and its variant LHC$_{\text{TC}}$-SE. Specifically, the model achieved a 27\% reduction in Mean Absolute Error (MAE) and a 33\% improvement in Median Absolute Error (MdAE) over baseline estimators, along with a 15\% increase in Standardized Accuracy (SA) when compared to advanced state-of-the-art models. Notably, the SBERT-LGBM variant optimized by Manual Tuning, which integrates Sentence-BERT for feature extraction and LightGBM for SP estimation, outperformed these existing SP estimators across 17 of the 26 projects, achieving statistically significant gains with large effect sizes in four critical cases.

The integration of SBERT's semantic analysis and the advanced handling of complex project data by GBT algorithms proved essential in reducing estimation errors, offering a robust decision-support tool for ASD project management. These results underscore the potential of automated SEE models to enhance project planning, resource allocation, and overall management efficiency. This study contributes a methodologically sound, scalable approach to SEE, setting a benchmark for further research into automated estimation models that adapt seamlessly within Agile landscapes.