Exploring Consistent Feature Selection for Software Fault Prediction: An XAI-based model-agnostic Approach

Numerous Feature Selection (FS) techniques have been widely utilized in Software Engineering (SE) to enhance the predictive accuracy of Machine Learning (ML) models. However, how consistently these FS techniques extract features under various data changes (made to the training data) remains underexplored. While prior studies have assessed the stability of traditional FS techniques (e.g., Information Gain, Genetic Search, etc.), their findings remain limited. With the growing use of eXplainable Artificial Intelligence (XAI) in SE, it is important to assess the level of consistency of model-agnostic FS techniques to ensure their reliability within dynamic learning environments. This study evaluates the consistency of Permutation Feature Importance (PFI) and SHapley Additive exPlanations (SHAP), across five ML models, i.e., Linear Regression(LR). Multi-layer Perceptron (MLP), Random Forest (RF), Decision Trees (DT), Support Vector Machines(SVM), on six Software Fault Prediction datasets under various validation methods (such as 3-fold, Bootstrap etc.), data normalization, and dataset modifications. The findings reveal that model-agnostic FS shows higher consistency than traditional FS techniques across all changes. In the case of validation-based consistency and using the SHAP, SVM and DT achieve the highest average feature consistency (100%), while MLP achieves the lowest (74.27%). Similarly, using PFI, LR, DT, and SVM achieves 100% consistency, whereas MLP remains the lowest consistency at 44.03%. In the case of data change-based consistency, using SHAP, MLP achieves the highest consistency (76.20%), whereas SVM has the lowest (70.98%). Using PFI, RF achieves the highest average consistency (77.24%), and MLP is the least consistent (44.93%). Similarly, in an overall comparison, both XAI-based techniques outperform traditional techniques, confirming their reliability for SFP tasks.