Performance Analysis of Deep Learning Models for Software Fault Prediction Using the BugHunter Dataset
Performance Analysis of Deep Learning Models for Software Fault Prediction Using the BugHunter Dataset
Abstract
Software fault prediction (SFP) involves the identification of potentially fault-prone modules before the testing
phase in the software development lifecycle. By predicting faults early in the development process, the SFP process enables software developers to focus their efforts on components that may contain faults, thereby enhancing the overall quality and reliability of the software. Machine learning and deep learning techniques have been widely applied to train SFP models. However, these approaches face several challenges, including irrelevant or redundant features, imbalanced datasets, overfitting, and complex model structures. The NASA dataset from the PROMISE repository is the most commonly used dataset for fault prediction. Recently, the BugHunter dataset with its substantially larger number of instances was explored to train the SFP models. In this study, we present the comparative study of three deep learning models, including Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) and four machine learning models as K-Nearest Neighbors (KNN), Multilayer Perceptron (MLP), Adaptive Boosting (AdaBoost),
Extreme Gradient Boosting (XGB) to investigate the performance of SFP models on the BugHunter dataset. We employ the Lasso method for feature selection and apply the Synthetic Minority Oversampling Technique (SMOTE) to address the issue of imbalanced data, aiming to enhance the accuracy of the results. The experimental findings reveal that CNN and RNN outperformed other machine learning models, achieving the best overall performance.
References
S. Najihi, S. Elhadi, R. A. Abdelouahid, and A. Marzak, “Software testing from an agile and traditional view,” Procedia Computer Science, vol. 203, pp. 775–782, 2022.
A. Balaram and S. Vasundra, “Prediction of software faultprone classes using ensemble random forest with adaptive synthetic sampling algorithm,” Automated Software Engineering, vol. 29, no. 1, p. 6, 2022.
M. Mangla, N. Sharma, and S. N. Mohanty, “A sequential ensemble model for software fault prediction,” Innovations in Systems and Software Engineering, vol. 18, no. 2, pp. 301–308, 2022.
S. S. Rathore and S. Kumar, “A study on software fault prediction techniques,” Artificial Intelligence Review, vol. 51, pp. 255–327, 2019.
S. K. Pandey, R. B. Mishra, and A. K. Tripathi, “Machine learning based methods for software fault prediction: A survey,” Expert Systems with Applications, vol. 172, p. 114595, 2021.
R. Ferenc, P. Gyimesi, G. Gyimesi, Z. Tóth, and T. Gyimóthy, “An automatically created novel bug dataset and
its validation in bug prediction,” Journal of Systems and Software, vol. 169, p. 110691, 2020.
C. Jin, S.-W. Jin, and J.-M. Ye, “Artificial neural networkbased metric selection for software fault-prone prediction
model,” IET software, vol. 6, no. 6, pp. 479–487, 2012.
S. Wang, T. Liu, J. Nam, and L. Tan, “Deep semantic feature learning for software defect prediction,” IEEE Transactions on Software Engineering, vol. 46, no. 12, pp. 1267–1293, 2018.
R. Malhotra, “A systematic review of machine learning techniques for software fault prediction,” Applied Soft Computing, vol. 27, pp. 504–518, 2015.
S. T. Cynthia, B. Roy, and D. Mondal, “Feature transformation for improved software bug detection models,” in Proceedings of the 15th Innovations in Software Engineering Conference, pp. 1–10, 2022.
C. Zhang, P. Patras, and H. Haddadi, “Deep learning in mobile and wireless networking: A survey,” IEEE Communications surveys & tutorials, vol. 21, no. 3, pp. 2224–2287, 2019.
H. K. Dam, T. Pham, S. W. Ng, T. Tran, J. Grundy, A. Ghose, T. Kim, and C.-J. Kim, “A deep tree-based model for software defect prediction,” arXiv preprint arXiv:1802.00921, 2018.
H. Liang, Y. Yu, L. Jiang, and Z. Xie, “Seml: A semantic lstm model for software defect prediction,” IEEE Access,
vol. 7, pp. 83812–83824, 2019.
A. V. Phan and M. Le Nguyen, “Convolutional neural networks on assembly code for predicting software defects,” in 2017 21st Asia Pacific Symposium on Intelligent and Evolutionary Systems (IES), pp. 37–42, IEEE, 2017.
A. B. Farid, E. M. Fathy, A. S. Eldin, and L. A. Abd- Elmegid, “Software defect prediction using hybrid model
(cbil) of convolutional neural network (cnn) and bidirectional long short-term memory (bi-lstm),” PeerJ Computer Science, vol. 7, p. e739, 2021.
Z. M. Zain, S. Sakri, N. H. Asmak Ismail, and R. M. Parizi, “Software defect prediction harnessing on multi 1-
dimensional convolutional neural network structure.,” Computers, Materials & Continua, vol. 71, no. 1, 2022.
L. Qiao, X. Li, Q. Umer, and P. Guo, “Deep learning based software defect prediction,” Neurocomputing, vol. 385, pp. 100–110, 2020.
R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 58, no. 1, pp. 267–288, 1996.
N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” Journal of artificial intelligence research, vol. 16, pp. 321–357, 2002.
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623–2631, 2019.
S. B. Imandoust, M. Bolandraftar, et al., “Application of k-nearest neighbor (knn) approach for predicting economic events: Theoretical background,” International journal of engineering research and applications, vol. 3, no. 5, pp. 605–610, 2013.
M.-C. Popescu, V. E. Balas, L. Perescu-Popescu, and N. Mastorakis, “Multilayer perceptron and neural networks,” WSEAS Transactions on Circuits and Systems, vol. 8, no. 7, pp. 579–588, 2009.
Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of computer and system sciences, vol. 55, no. 1, pp. 119–139, 1997.
T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794, 2016.
J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, et al., “Recent advances in convolutional neural networks,” Pattern recognition, vol. 77, pp. 354–377, 2018.
I. Sutskever, J. Martens, and G. E. Hinton, “Generating text with recurrent neural networks,” in Proceedings of the 28th international conference on machine learning (ICML-11), pp. 1017–1024, 2011.
G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5929–5955, 2020.
