
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39266651
72307
10.1038/s41598-024-72307-0
Article
Detecting refactoring type of software commit messages based on ensemble machine learning algorithms
Al-Fraihat Dimah d.fraihat@iu.edu.jo

1
Sharrab Yousef 2
Al-Ghuwairi Abdel-Rahman 3
Sbaih Nour 3
Qahmash Ayman 4
1 https://ror.org/04d4bt482 grid.460941.e 0000 0004 0367 5513 Department of Software Engineering, Faculty of Information Technology, Isra University, Amman, 11622 Jordan
2 https://ror.org/04d4bt482 grid.460941.e 0000 0004 0367 5513 Department of Data Science and Artificial Intelligence, Faculty of Information Technology, Isra University, Amman, Jordan
3 https://ror.org/04a1r5z94 grid.33801.39 0000 0004 0528 1681 Department of Software Engineering, Faculty of Prince Al-Hussien Bin Abdallah II for Information Technology, The Hashemite University, Zarqa, Jordan
4 https://ror.org/052kwzs30 grid.412144.6 0000 0004 1790 7100 Department of Information Systems, King Khalid University, Abha, Saudi Arabia
12 9 2024
12 9 2024
2024
14 2136715 8 2023
5 9 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Refactoring is a well-established topic in contemporary software engineering, focusing on enhancing software's structural design without altering its external behavior. Commit messages play a vital role in tracking changes to the codebase. However, determining the exact refactoring required in the code can be challenging due to various refactoring types. Prior studies have attempted to classify refactoring documentation by type, achieving acceptable results in accuracy, precision, recall, F1-Score, and other performance metrics. Nevertheless, there is room for improvement. To address this, we propose a novel approach using four ensemble Machine Learning algorithms to detect refactoring types. Our experimentation utilized a dataset containing 573 commits, with text cleaning and preprocessing applied to address data imbalances. Various techniques, including hyperparameter optimization, feature engineering with TF-IDF and bag-of-words, and binary transformation using one-vs-one and one-vs-rest classifiers, were employed to enhance accuracy. Results indicate that the experiment involving feature engineering using the TF-IDF technique outperformed other methods. Notably, the XGBoost algorithm with the same technique achieved superior performance across all metrics, attaining 100% accuracy. Moreover, our results surpass the current state-of-the-art performance using the same dataset. Our proposed approach bears significant implications for software engineering, particularly in enhancing the internal quality of software.

Keywords

Refactoring
Ensemble boosting machine learning
Documentation
Commits classification
Subject terms

Computer science
Software
The Deanship of Scientific Research at King Khalid UniversityRGP.1/209/43. Qahmash Ayman issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

In the software engineering field, refactoring refers to the process of enhancing the internal structure of a program without altering its external behavior1. Realizing how refactoring is used in software systems may assist in software maintenance and evolution, as well as understanding effective application design principles and improving code comprehension2. Further, refactoring can be used to remove “code smells” which are bad codes or bad designs that degrade software quality in terms of maintainability, reusability, and understandability3. Moreover, refactoring can be “technical debt” which refers to a problem that makes the software development process difficult and necessitates a significant amount of effort and time to continue development, such as a lack of documentation, a lack of software tests, or security issues4.

Detecting refactoring is a challenging task since refactoring procedures are not usually documented by developers. Further, refactoring activities are frequently performed in conjunction with other modifications, making it considerably more difficult to extract them out of complicated code changes5. Therefore, detecting the requested refactoring for a file, class, or any part of the software system can aid in understanding and applying the requested change correctly either by detecting the refactoring location, refactoring type, or refactoring history for a specific part of the system. However, there are many refactoring issues raised in manual tracking systems6. For example, these tracking systems include a huge number of reported issues, hence it’s difficult and time-consuming to manually determine which of these activities are refactoring-based activities or not. A significant amount of research has focused on assessing developers’ refactoring techniques by mining code changes and commit messages.

The use of commit messages for identifying reasons behind developers’ decisions to refactor and optimize various structural metrics, including coupling and complexity, as well as quality attributes such as readability has been explored by researchers1. This analysis relies on the assumption that refactoring is described in a way that allows for differentiation between them (e.g., different descriptions are used for Rename method and Move method). Therefore, it is important to understand the accuracy of this assumption and how refactoring can be classified based on their descriptions in commit messages. In addition, commit messages have been utilized for recommending refactoring operations, and several studies have developed taxonomies and approach for identifying and labeling refactoring types from source code or documentation. However, little is known about the accuracy of descriptions of refactoring in commit messages and there is a need for improving their reliability7,8.

In this study, we utilize four ensemble ML algorithms including AdaBoost, CatBoost, RUSBoost, and XGBoost for detecting refactoring types and to train the models. We used a refactoring documentation dataset that includes 573 commits. Text cleaning and prepossessing were applied to the commits’ description text were conducted to solve data imbalances and multi-class data issues found in the dataset. Several techniques were used to improve accuracy, these techniques are non-hyperparameter, hyperparameter optimization using GridSearchCV, feature engineering using TFIDF, feature engineering using bag-of-words, binary transformation using one-vs-one, and binary transformation using one-vs-rest classifiers. Several experiments were conducted to evaluate the resulting models and select the optimal model. The metrics used to compare the performance of the models are accuracy, precision, recall, f1-score, Macro average, and weighted average. The results of this study reveal that the experiment conducted with feature engineering using the TF-IDF technique outperformed the other techniques. For the same technique, the performance of the XGBoost model outperformed the other ensemble ML boosting models with the amount of 100% for all the metrics used compared with state-of-the-art which achieved 89.60% and 76.10% for the precision and the recall for the same dataset.

This paper is organized as follows: "Background and related work" provides the research background and reviews the literature and related work, "Research methods and the proposed approach" presents the proposed approach and research methods, "Building experimental classification models" discusses the conducted experiments, "Results and analysis" presents the results obtained, and "Summary of results and discussion" provides a summary of the results. Finally, "Comparison with previous work" concludes the study and outlines limitations and future work.

Background and related work

In the field of software engineering, a commit message refers to a summary of the modifications performed on the source code files during a commit. A commit represents a snapshot of the code change made at a moment and it is commonly accompanied by a commit message that outlines the intent behind these changes9. Commit messages serve several purposes. They provide a summary of the code changes to help developers track and review modifications over time. In addition, they facilitate collaboration among team members. Commit messages also help in the identification and resolution of bugs and improve the documentation of the codebase. A commit message should be brief, descriptive, and easily comprehensible. It should offer context for other developers to grasp the purpose of the changes while adhering to a consistent format or style that ensures readability and maintains consistency across the codebase9. In software development, it is essential to have concise commit messages. A good commit message would be something like “Fixed a bug in the login form validation,” “Implemented support for the feature X,” or “Optimized code to enhance performance.” On the contrary, poor commit messages tend to be ambiguous, unclear, or excessively lengthy, making it challenging for others to comprehend the purpose of the changes. Writing commit messages adheres to best practices in software development and contributes to better code quality and maintainability9.

Just like the code itself, commit messages can become unclear and difficult to maintain over time. As the codebase evolves and grows, it may become necessary to refactor the commit messages to improve their quality and maintain-ability. Refactoring refers to different techniques or methods that are used to improve the internal quality of software systems without changing their external behavior1. It is a process of restructuring the code of a software system to make it more readable, maintainable, and extensible. Refactoring types may include methods for improving the organization and structure of code, such as renaming variables, extracting methods, or creating classes, among others10. There are types of refactoring, each serving a unique purpose and offering specific benefits. The choice of the suitable type depends on the specific requirements of the software system being developed or maintained. Some used forms of refactoring include “Extract Method”, “Inline Method”, “Move Method”, “Rename Method”, “Extract Variable”, “Replace Magic Number”, “Consolidate Conditional Expression”, “Remove Dead Code”, and “Introduce Parameter Object”11. By regularly refactoring code, developers can improve its structure, readability, and performance, making it easier to maintain and develop over time.

Ensemble boosting machine learning algorithms

Ensemble boosting algorithms in machine learning are methods where multiple weak classifiers are joint together to create a strong classifier. The concept of ensemble refers to “a machine learning technique that combines several base classifiers in order to produce one optimal predictive model and enhances the prediction accuracy” whereas boosting indicates the approach that combines these classifiers12.

This study aims to classify refactoring types utilizing machine learning algorithms such as, AdaBoost, CatBoost, RUSBoost, and XGBoost. These boosting algorithms utilize trained decision trees on different subsets of data. To help the weak classifiers learn from their errors and improve the ensemble’s overall performance, the emphasis throughout model training is placed on correcting samples. By combining the predictions made by each decision tree, the ensemble algorithm would be capable of making more precise predictions compared to a single decision tree acting alone. A brief overview of each algorithm is introduced as follows.

Adaptive boosting, referred to as AdaBoost, is a popular machine learning algorithm that was developed by Yoav Freund and Robert Schapire in 1996 and is utilized for classification tasks13. The AdaBoost technique combines several weak or simple classifiers to produce a powerful, robust classifier. These weak classifiers are trained using modified training sets that provide more weight to samples that the prior weak classifier incorrectly classified. AdaBoost has the potential to assist in determining which areas of a codebase are more vulnerable to errors or require restructuring in the context of software refactoring. Finding places that need testing, code review, or refactoring might be improved by performing this. The first step in using AdaBoost in software refactoring is to find a set of features that are relevant to the work at hand. These characteristics may include measurements for code churn, code complexity, or related factors14. Data on code defects or past refactoring are utilized to train an AdaBoost model. This model could subsequently predict which parts of the code are more likely to be flawed or need refactoring. The model can aid in prioritizing testing, code review, and refactoring tasks once it has been trained. For instance, testing resources can be assigned if the model indicates that a certain module is likely to have flaws. Similar to this, if the model predicts that a certain module will need refactoring, development resources can be assigned to it, or it can be given higher priority in the development backlog15.

CatBoost, also known as Categorical Boosting, is a machine learning algorithm that is widely used for both classification and regression tasks16. CatBoost was developed in 2017 by Yandex, a Russian technology company. One of the significant features of CatBoost is the ability to handle categorical features effectively. Unlike traditional gradient boosting algorithms that typically require numerical encoding for features, CatBoost can directly handle them without explicit feature encoding. In software refactoring, CatBoost can prove to be valuable by predicting which parts of a codebase are more likely to change or need refactoring in the future. This prediction capability assists in the determination of areas that need attention and prioritizing refactoring efforts16. To utilize CatBoost for the software refactoring, the first step is to determine the set of relevant features for the task. These may encompass measurements metrics of code complexity, frequency of changes made to files or modules (i.e., code churn) or other relevant factors. Next, historical data on code changes are employed to train the CatBoost model which predicts the parts of the code that will most likely get changes. Once the model has been trained, it becomes possible to prioritize refactoring efforts and pinpoint areas in the codebase that need further testing or scrutiny16.

RUSBoost, also known as Random Under Sampling Boosting, is a machine learning algorithm introduced by Seiffert in 2010 that incorporates two methods: Random Under Sampling (RUS) and AdaBoost. It can tackle the issue of imbalanced classification problems as initially17. Imbalanced classification poses a challenge in software refactoring, where the quantity of code elements requiring refactoring is significantly less than those that don’t. An approach for addressing this issue is to employ RUSBoost, which entails balancing the training data prior to model training and combining the results with AdaBoost. Utilizing a random subset of the training data’s majority class examples, that is code segments that don’t require refactoring. In order to balance the training data, RUSBoost randomly selects a fraction of instances from the majority class (i.e., code components that do not require refactoring) that matches the number of minority class examples (i.e., code components that do require refactoring). Then, AdaBoost is applied to train a classifier on this balanced dataset18. RUSBoost is a helpful tool for detecting which areas of the code require refactoring. The initial step in using RUSBoost is to identify the appropriate features. Code complexity, code churn, and other significant metrics are among these features. The RUSBoost model can be trained to predict where specific areas of the code need to be refactored using historical data on code refactoring. Then it is feasible to decide which refactoring tasks should be prioritized and identify the parts of the codebase that require attention16.

XGBoost, also known as Extreme Gradient Boosting, is a widely used machine learning algorithm introduced by Tianqi Chen in 2016 which is utilized for both classification and regression tasks19. XGBoost combines multiple decision trees to reduce errors. It implements an approach to dividing nodes within each tree and can handle missing data without the need for imputation. XGBoost can be utilized to forecast the areas in a codebase that’re prone to defects or would need refactoring20. To utilize XGBoost for software refactoring purposes, the set of related features has first to be identified. Next, XGBoost model needs training on code defects and previous refactoring. This model would then be used to predict which parts of the code are most likely to have defects or require refactoring. Once the model is trained, testing efforts, code reviews, and refactoring tasks can be prioritized accordingly20. For instance, in case the model anticipates a specific module that have defects, extra testing resources can be allocated to that module to review the code. Similarly, if the model predicts that a particular module may need refactoring, development resources can be assigned to that module and give it higher priority16. Overall, all these boosting algorithms are powerful tools for building ML models and have unique strengths and weaknesses depending on the specific problem and data being analyzed21,22. Table 1 provides an overview of the algorithmic approaches, mathematical formulations, time complexities, strengths, and weaknesses of AdaBoost, CatBoost, RUSBoost, and XGBoost.Table 1 Detailed algorithmic approach utilized in this research for ensemble learning models.

Algorithm	Approach	Mathematical formulation	Weaknesses	Training complexity	Inference complexity	Strengths	
AdaBoost	Combining multiple weak classifiers to create strong classifiers and assigning higher weights to misclassified instances	Minimizes exponential loss function and updates weights of misclassified instances iteratively	Sensitive to noisy data, overfitting with many iterations	O (T. n. d)	O (T. d)	Focuses on hard cases, improves weak classifiers	
CatBoost	Handling categorical features without extensive pre-processing. Using ordered boosting to minimize overfitting	Combines gradient boosting on decision trees with ordered target statistics for categorical features	Can be slower with very large datasets	O(T. n. log n)	O (T. log n)	Efficient with categorical data, reduces overfitting	
RUSBoost	Combining randomly under-sampling of the majority classes with boosting techniques to address class imbalance	Integrates under-sampling with AdaBoost, updating weights and randomly sampling the majority class in each iteration	May lose information due to under-sampling	O (T. nmin. d)	O (T. d)	Addresses class imbalance, robust against overfitting	
XGBoost	Optimized gradient boosting framework that uses regularized model to control overfitting to improve performance	Optimizes a regularized objective function combining the training loss and regularization terms	Requires careful parameter tuning	O (T. n. log n)	O (T. log n)	High performance, controls overfitting	

Optimization approaches for machine learning

The primary objective of ML is to develop a model that operates well and predicts accurately in a certain set of instances. ML optimization is needed to do this. The optimization approaches employed in this work are hyperparameter optimization, feature engineering, and binary transformation. These approaches are briefly discussed below.

The task of selecting a set of ideal hyperparameters for a learning algorithm is known as hyperparameter optimization or tuning. A hyperparameter is a parameter whose value controls the learning process and describes the structure of the model21. There are ways to adjust hyperparameters, and it is recommended that a significant portion of this process focuses on adjusting the learning rate, which is considered the most crucial hyperparameter. However, it is not possible to determine the ideal learning rate for a specific model and dataset through mathematical analysis. Instead, it is necessary to experiment with various learning rates until an acceptable one is found. The standard learning rate values are 0.1 or 0.0121. Another hyperparameter used in ML algorithms is the n estimator. It determines the number of decision trees to use in the ensemble22. Increasing the value of the n estimator can improve the performance of the model, but it also increases computational cost and memory usage. It’s important to find the optimal value of n estimators that balances the trade-off between model performance and computational efficiency.

The second optimization approach is the feature engineering which is “the act of choosing, altering, and converting raw data into features that may be utilized in supervised learning”23. To allow ML to function effectively and successfully on new tasks, better features may need to be designed and trained. A “feature” is any quantifiable input that may be included in a prediction model23. Simply said, feature engineering is the process of transforming raw observations into desired features using the approaches of ML. In this approach, we use two of the most commonly used feature engineering techniques, especially for text classification. These techniques are TF-IDF and Bag-of-Words.

TF-IDF (Term Frequency-Inverse Document Frequency) is a feature engineering technique used in Natural Language Processing (NLP) that measures the relevance of a word in a document corpus. It calculates the term frequency (TF), which is the number of times a word appears in a document and scales it by the inverse document frequency (IDF), which measures how common or rare the word is across all documents in the corpus23. The score obtained indicates the significance of the word within the given document. TF-IDF can be applied in different natural language processing (NLP) tasks, including text categorization, information retrieval, and recommendation systems to represent textual data as features and metrics for machine learning models24. In contrast, the Bag of Words (BoW) technique is widely used in NLP to transform textual data into a collection of features that can be utilized in ML models25. The bag of words approach includes representing the document as a “bag” of words disregarding their arrangement and context and focusing solely on their frequency. This method is powerful for extracting features from text data extensively employed in different NLP tasks like sentiment analysis, text classification, and topic modelling. Nevertheless, disregarding word order and context may result in high dimensional and sparsely populated feature vectors when dealing with large vocabularies25.

Binary transformation is an alternative method for optimization. This technique involves converting categorical or numerical features into binary form, where each observation is represented as either 1 or 0. This representation is determined by whether the observation meets a condition or not26. It is useful in capturing non-linear relationships and converting categorical features into binary features. However, the choice of threshold values must be carefully considered to avoid introducing bias or overfitting. One-vs-Rest (OvR) and One-vs-One (OvO) are two binary transformation techniques used in multi-class classification problems. OvR trains a separate binary classifier for each class against the rest of the classes, while OvO trains a separate binary classifier for each pair of classes. They are commonly used in ML algorithms that do not natively support multi-class classification. However, they may have limitations, such as requiring more training time and potentially introducing biases in the classification results26.

Related work

This paper focuses on ML approaches to detect refactoring and classify them. The ML approaches have been explained previously. Thus, in this part, we are interested in exploring the related research that employed ML algorithms for refactoring classification.

The study of Marmolejos et al.27 provided an approach for extracting features from commit messages using chi-squared or feature hashing, preprocessing text, and supervised ML (Bayes’s point machine, averaged perceptron, logistic regression, boosted decision tree, and neural network) to binary classify the refactoring documentation as refactoring or not. Chi-Squared and Fisher scores with Bayes point machines provided a very good performance with accuracy, and the F1-score reaches up to 96%.

Another study for extracting refactoring type for commit message based on ML approach introduced by AlOmar et al.5. First, they preprocessed commit messages using NLP. Then, they extracted features such as the most important keywords that support the training process using N-gram. They have used nine ML algorithms to classify these commits into six categories at the method level. As a result, the accuracy reaches up to 93 for some data classes. However, the models show bad behavior for other data classes with an accuracy equal to 45%.

Nyamawe proposed an approach that included two classification methods. The first was a binary classification of refactoring and non-refactoring commits28. The second one was multilabel classification, which categorizes the refactoring commit message into its related refactoring type. The model performed well for binary classification, with an F-measure of up to 87.56%. The model provided the best performance for multi-label classification, with accuracy reaching up to 80.86%.

AlOmar et al.29 provided a machine-learning approach for classifying commits for 800 Java projects into three classifications, namely internal quality attributes, external quality attributes, and code smell resolution. They stated that several factors may lead to refactoring, not only having code smells. Also, they stated that the developers used a specific way to describe and document the refactoring activity; these can be used to understand the way that developers write the refactoring commit description and document their work.

Another study presented by Aniche30 for detecting refactoring types using six ML algorithms. They have used 2 million refactorings on 11,149 real projects. The RandomForest model achieved an accuracy of up to 90%. In addition, they found that the performance of training models can be enhanced when using datasets from many sources. However, Sheneamer31 introduced an approach that automatically detects the needed refactoring type based on features extracted from the selected ML algorithms and defined code clones. Furthermore, the author improved training model performance by identifying and excluding outliers as unknown clones. The accuracy of the training models was very high and acceptable, reaching up to 99.5%.

In addition, Krasniqi and Cleland-Huang32 proposed two phases of crime classification using ML algorithms. First, they classified commits into two categories: those Containing Refactoring Information (CMR) and those not containing refactoring information (non-CMR). They classified CMR commits into 12 refactoring types. According to their study, the best performance achieved for binary classification was with an F1-score reaching up to 84%, and SVM showed the best performance for multiclass classification with an F1-score reaching up to 71%.

In the study of Nyamawe et al.33, the researchers used 43 open-source Java projects to train ML models to detect the recommended refactoring. They used binary and multi-label classifiers. First, they classify the commits as refactoring or non-refactoring. The type is then determined for each refactoring. The obtained F1-measure totals 74.63%. The research conducted by Gharbi et al.34 has created a multi-label classification model using binary relevance with logistic regression to classify refactoring commits into three maintenance activities. They achieved an F1 measure of 45.79%.

The study proposed by Fontana et al.35 carried out a number of experiments on detecting code smells. They have used four datasets from 74 software systems. Each data file includes instances of the same code with a true or false value. The chosen code smells are Data Class, God Class, Feature Envy, and Long Method. Several ML algorithms have been used. They have achieved acceptable accuracy of approximately 96% for each dataset file. In addition, they stated that the algorithms showed good performance when cross-validation datasets were used. Furthermore, Nucci et al.36 applied the same approach as in the previous study, but they used a different dataset that includes more than one code smell type for each dataset file.

Levin and Yehudai37 have gathered the commits dataset and labeled it manually. Then, they classified these tasks into three activities: preventive, corrective, and adaptive, using three ML models: Gradient Boosting Machine (GBM), J48, and RandomForest. They stated that the RandomForest model showed the best performance, with accuracy reaching up to 76%. However, Rodriguez et al.38 provided a clustering approach that delivered a list of refactoring suggestions as clusters. They have used non-refactored WSDL documents to extract the data. They preprocessed the data. Then they passed it through clustering algorithms. The COBWEB clustering algorithm provided the best performance for the selected dataset. An interesting study of Atwi et al.39, which used the same dataset as used in our study, presented an automated approach called PYREF that automatically classifies refactoring documentation based on its type. The researchers have used a dataset that includes 9 data classes and 573 refactoring commits. PYREF achieved 89.60% and 76.10% for the precision and the recall metrics, respectively.

Research methods and the proposed approach

This section introduces the methods used in this research and the proposed approach for categorizing refactoring commits. This approach is implemented using Python. Since the refactoring commits are textual data, NLP preprocessing is needed. Hence, text cleaning and prepossessing steps are applied to the commit description text. The data is divided into two datasets: training and testing. To simplify data, reduce memory usage, and make it compatible with the adopted ML algorithms, label encoding is used to assign numerical labels to refactoring types. Four ensemble machine algorithms were selected as per the recommendation of49 due to the existence of imbalanced data and multi-class issues in our dataset. Figure 1 shows the data and their distribution on the refactoring type labels.Fig. 1 The distribution of refactoring commits based on their types.

The four classification algorithms are AdaBoost, CatBoost, RUSBoost, and XGBoost. Several techniques are used to enhance the classifier’s performance. These techniques are hyperparameter optimization using GridSearchCV, feature engineering using TF-IDF, feature engineering using bag-of-words, binary transformation using one-vs-one, and binary transformation using one-vs-rest classifiers.

Our approach involved several steps to ensure that the experimentation is as thorough as possible, and to select optimal model. First, the dataset was chosen and after that text data cleaning and text preprocessing were performed. Secondly, data were split, and the labels were encoded. We selected four machine learning algorithms for our experiments: AdaBoosting, CatBoost, RUSBoosting, XGBoost. For each algorithm, we tested six different setups including: no hyperparameter tuning, optimal hyperparameter tuning, feature extraction with TF-IDF, feature extraction bag of words, binary conversion one-vs-one and binary conversion one-vs-rest. Each model was trained and evaluated under these setups, and we repeated the process for all 24 experiments. Lastly, we methodically compared the outcomes in order to identify the most suitable model. Figure 2 provides a summary of the proposed approach.Fig. 2 The proposed approach.

Dataset description

The dataset that is used in this study is a refactoring documentation dataset for these projects: DIT6, TEXAR7, and FFMPEG-PYTHON839 and includes 573 commits. A commit is a specific file modification in the code repository40. Each commit is labeled with its refactoring type. In the dataset, there are nine commit labels: “Add Parameter”, “Rename Method”, “Move Method”, “Change/Rename Parameter”, “Remove Parameter”, “Extract Method”, “Pull Up Method”, “Inline Method”, and “Push Down Method”. Table 2 provided definitions for each data label in our dataset.Table 2 Refactoring type labels and their descriptions in the dataset.

Refactoring type	Description	
Adding parameter	This refactoring type adds a new parameter to a method when there is insufficient data to apply the method’s methodology41	
Renaming method	This refactoring type changes the name of a method when its name does not describe the method’s functionality42	
Moving method	This refactoring type involves moving a part of the code from a method in a specific class into a newly created method in another class that uses the method the most31	
Changing/renaming parameter	This refactoring type changes the name of a parameter when the parameter’s name does not describe its use43	
Removing parameter	This refactoring type removes a parameter when it is no longer used in the code44	
Extracting method	This refactoring type creates a new method that uses an extracted piece of code as its input, replacing the original code fragment. The extract technique is divided into parts31	
Pulling up method	This refactoring type generalizes a method that is frequently used in many files in the code by moving it to a shared superclass31	
Inlining method	This refactoring type replaces instances with the original method declaration and adds an extension method within a single statement body that calls the original declaration45	
Pushing down method	This refactoring type involves transferring a method from a superclass to a subclass that is only and frequently used45	

Data cleaning and preprocessing

Since the dataset utilized in this study is text-based, NLP and the Natural Language Toolkit (NLTK) were used to clean it up so that it can be easily included into our models and produce acceptable accuracy. For the cleaning process, the TensorFlowKeras-Library was used to call the required Python methods46–48 as follows:Convert the text to lowercase: to make the text smoother and maintain consistency by converting to lowercase letters. In this step, we use lower() python function.

Remove punctuation: all punctuation in the text is replaced with white spaces.

Remove extra spaces: punctuation removal could leave more white space behind. By including a validation Python statement that substitutes additional spaces with just one space, all double white spaces are eliminated.

Word lemmatization: all suffixes and prefixes of words on the dataset are removed and only the base of the word is returned.

Remove stop words: a list of stopwords is applied to the dataset, resulting in their exclusion, in order to remove irrelevant or insignificant words.

Figure 3 shows an example of a sentence prior to and after using the prepossessing procedures. The second sentence is the outcome of data processing techniques. As shown in the figure, the second sentence is clearer and easier to pass via ML algorithms.Fig. 3 Commit description before and after applying data prepossessing techniques.

Evaluation metrics

To evaluate the proposed model’s performance, various measures have been employed, including accuracy, precision, recall, F1-score, as well as multi-class metrics such as macro average and weighted average.Accuracy is a measure used to determine how many records in the dataset are correctly classified by the model. It is calculated by dividing the sum of correctly predicted rows by the total number of records in the dataset49.

1 Accuracy=TP+TNTP+FP+FN+TN

where TP, TN, FN stand for true positive, true negative, and false negative, respectively.Precision is an effective model evaluation metric used to measure the accuracy of detecting relevant items for a certain task. It is beneficial to deal with classification issues such as imbalanced classes where the number of instances in different classes is unequal. To find the precision value, the number of positive instances is divided by the total number of positive instances detected by the model. Similarly, “precision assesses the ratio of correctly identified positives out of all predicted positives”49.2 Precision=TPTP+FP

where TP, TN, FN stand for true positive, true negative, and false negative, respectively.

Recall serves as an evaluation metric to assess how well can the model identify all the relevant components. It is determined by “dividing the number of true positives by the total count of relevant elements.” It calculates the ratio of true positives within the entire dataset’s relevant elements49.3 Recall=TPTP+FN

where TP, TN, FN stand for true positive, true negative, and false negative, respectively.

F1-score is a commonly used metric in model evaluation to assess the correctness of test data50.4 F1-score=2×Precision×RecallPrecision+Recall

Macro average is a multi-class metric obtained by computing the mean for each data class’s precision, recall, and F1-score measures50,51.

Weighted average is a multi-class metric evaluated similarly to the macro average, with one difference; it considers the frequency of each class known as the support value50,51.

Building experimental classification models

This research aims to classify refactoring commits to detect the related refactoring type and build an efficient model to predict the type of a new commit correctly. A refactoring documentation dataset was used consisting of 573 commits for nine refactoring types mentioned in Table 2. To achieve this, text cleaning and preprocessing were applied to the commit description text. Data were split into training and testing datasets. Label encoding was used for the types of refactoring. Four ensemble-boosting machine-learning algorithms have been used to train the models. These algorithms are the AdaBoost, CatBoost, RUSBoost, and XGBoost algorithms. Several experiments were conducted based on hyperparameter optimization using GridSearchCV, feature engineering using TF-IDF, feature engineering using bag-of-words, binary transformation using one-vs-one, and binary transformation using one-vs-rest classifiers. The following clarifies how each experiment is built.Non-Hyperparameter Experiment, in this experiment, before applying any enhancement algorithm, we aim to investigate the performance of the four ensembling algorithms without tuning any hyperparameter. We used fixed hyperparameters as ground truth for the other experiments. Each of the four algorithms (i.e., AdaBoost, CatBoost, RUSBoost, and XGBoost) has its own set of parameters, such as n estimator, learning rate, and the number of epochs, that aims to enhance its performance based on the values of these parameters and the input dataset. Table 3 shows the values of n estimators and the learning rate for each algorithm before tuning.

Finding Optimal Hyperparameter Experiment, this experiment is conducted to find the optimal hyperparameters. This approach focuses on two parameters, as follows: n estimators and learning rate52. First, we tuned the two parameters for each algorithm using GridSearchCV instead of a trial-and-error process to save time and obtain more accurate values. GridSearchCV53 is a method provided by the Sklearn Python library54 that aims to try out several values, which we provide as a list of hyperparameters, with an array for each hyperparameter, and then the method chooses the one with the highest score. Additionally, we selected the value for n estimator in the range 1 to 50 and 0.01 to 1 for learning rate. Then, as a result of applying the GridSearchCV, we reach the best hyperparameter for each boosting algorithm for the used dataset. Table 4 shows the best values of n estimators and learning rate for each algorithm.

Table 3 Default values for non hyperparameters for each algorithm.

Algorithm	n estimators	Learning rate	
AdaBoost	50	1.0	
CatBoost	1000	0.03	
RUSBoost	50	1.0	
XGBoost	100	0.3	

Table 4 Optimal hyperparameter for each algorithm.

Algorithm	n estimators	Learning rate	
AdaBoost	47	0.27	
CatBoost	45	0.57	
RUSBoost	49	0.47	
XGBoost	48	0.58	

The following figures from Figs. 4, 5, 6 and 7 illustrate the best values for n estimators and the learning rate for each classification algorithm. For the AdaBoost algorithm, Fig. 4a shows that the n estimator with the value 47 provided the highest mean test score of 0.44. Figure 4b shows that the learning rate with the value 0.27 provided the highest mean test score of 0.44.Fig. 4 The performance of the AdaBoost hyperparameters.

Fig. 5 The performance of the CatBoost hyperparameters.

Fig. 6 The performance of the RUSBoost hyperparameters.

Fig. 7 The performance of the XGBoost hyperparameters.

For the CatBoost algorithm, Fig. 5a shows that the n estimator with the value 45 provided the highest mean test score of 0.79, and Fig. 5b shows that the learning rate with the value 0.57 provided the highest mean test score of 0.79. Regarding the RUSBoost algorithm, Fig. 6a shows that the n estimator with the value 49 provided the highest mean test score of 0.33. Figure 6b shows that the learning rate with the value 0.47 provided the highest mean test score of 0.33. Finally, the XGBoost algorithm shows that the n estimator with the value 48 provided the highest mean test score of 0.81 as shown in Fig. 7a. Furthermore, Fig. 7b shows that the learning rate with the value 0.58 provided the highest mean test score of 0.81.

Two common feature engineering techniques used in text classification are TF-IDF and Bag-of-Words.TF-IDF is a useful method in text classification that combines two measurement metrics: term frequency and inverse document frequency for feature extraction from the textual data. The frequency of a term is calculated by counting how frequently each phrase appears in the records. Conversely, the inverse document frequency is obtained by dividing the number of dataset records by the number of records that include the phrase55.

Bag of Words is a technique that is frequently employed for feature engineering. It’s an effective way to extract features from text. This approach converts text documents into vector representations, where each vector signifies the occurrence of terms in the document’s vector space. By doing this, each document is transformed into a vector that can be utilized for purposes such as text classification55,56.

Binary transformation is a method applied in handling multiclass classification problems. There are two types of binary transformation techniques known as One versus One (OvO) and One versus Rest (OvR)57,58.Binary Transformation One-versus-One (OvO) is “a binary transformation technique that entails fitting one classifier per class pair”. The class with the most votes is chosen at prediction time.

Binary Transformation One-versus-Rest (OvR) or One-versus-All is “a binary transformation technique that involves training a single classifier for each class”. Each classifier is trained on samples from one class and all the remaining samples combined.

It is worth mentioning that the NLTK tokenizer was used in all the experiments to convert text into tokens so that the data could be passed through classification algorithms. The next section introduces the results obtained from running the experiments.

Results and analysis

This research focuses on creating a model that can accurately classify refactoring commits by their type. A dataset of 573 commits for nine refactoring types was used, and text cleaning and preprocessing were applied to the commit descriptions. The data was split into training and testing sets, and label encoding was used for the types of refactoring. Four ensemble boosting ML algorithms were used, and several experiments were conducted to optimize hyperparameters and feature engineering using different techniques such as TF-IDF and bag-of-words. Additionally, two different binary transformation classifiers (one-vs-one and one-vs-rest) were used in the experiments. A total of twenty-four experiments, obtained from running six setups (techniques) on four algorithms, were conducted to determine which one performs best on the dataset of software refactoring commits. The results of running the experiments are compared. This section provides an evaluation and comparison of each experiment.

Results of non hyperparameter experiment

This experiment aimed at finding the exact behavior of the ensemble ML algorithm without tuning hyperparameters or using enhancement techniques. As a result, we achieved good accuracy using the XGBoost and CatBoost algorithms, with accuracy equal to 81% for CatBoost and 83% for the XGBoost algorithms. However, the RUSBoost and AdaBoost algorithms show a very low accuracy of 26% and 37%, respectively. Table 5 shows the evaluation metrics for ensemble algorithms in this experiment. Figure 8 summarizes the experimental results.Table 5 Non hyperparameter experiment.

Algorithm	AdaBoost	CatBoost	RUSBoost	XGBoost	
Accuracy	0.37	0.81	0.26	0.83	
Precision	0.18	0.79	0.28	0.75	
Recall	0.23	0.62	0.19	0.63	
F1-score	0.37	0.81	0.26	0.83	
Macro average	0.16	0.67	0.19	0.66	
Weighted average	0.23	0.80	0.30	0.82	

Fig. 8 Evaluation results of non-hyperparameter experiment for the 4 ensemble ML algorithms.

Results of finding optimal hyperparameter experiment

This experiment aims to find the optimal hyperparameters for each ensemble ML algorithm. The GridSeachCV technique is used to find optimal hyperparameters. The results of this experiment are shown in Table 6; we obtained similar results to the previous experiment. Figure 9 summarizes the experimental results.Table 6 Optimal hyper parameter experiment.

Algorithm	AdaBoost	CatBoost	RUSBoost	XGBoost	
Accuracy	0.37	0.81	0.16	0.80	
Precision	0.20	0.74	0.13	0.73	
Recall	0.24	0.63	0.11	0.62	
F1-score	0.37	0.81	0.16	0.80	
Macro average	0.16	0.66	0.03	0.64	
Weighted average	0.25	0.81	0.05	0.80	

Fig. 9 Evaluation results of finding optimal hyperparameter experiment for the four algorithms.

Results of feature engineering TF-IDF experiment

This experiment shows the best performance compared to the others. In the feature engineering using TF-IDF experiment, XGBoost shows an accuracy that reaches up to 100%. In addition, CatBoost shows an accuracy, reaching up to 99%, and AdaBoost accuracy equals 95%. However, RUSBoost shows low performance with 58 accuracy. Table 7 and Fig. 10 summarize the results of the feature engineering experiment using TF-IDF.Table 7 Feature engineering—TF-IDF vectorization experiment.

Algorithm	AdaBoost	CatBoost	RUSBoost	XGBoost	
Accuracy	0.95	0.99	0.58	1.00	
Precision	0.87	1.00	0.46	1.00	
Recall	0.83	0.94	0.38	1.00	
F1-score	0.95	0.99	0.58	1.00	
Macro average	0.84	0.96	0.39	1.00	
Weighted average	0.94	0.99	0.57	1.00	

Fig. 10 Evaluation results of feature engineering TF-IDF experiment for the four algorithms.

Results of feature engineering bag-of-words experiment

In the Bag-of-Words experiment, the performance of XGBoost and CatBoost was similar to the TF-IDF experiment, but the accuracy of the AdaBoost algorithm dropped to 55%. The RUSBoost algorithm still has a poor accuracy rate of up to 35%. Table 8 and Fig. 11 show the results.Table 8 Feature engineering—bag of word experiment.

Algorithm	AdaBoost	CatBoost	RUSBoost	XGBoost	
Accuracy	0.55	0.99	0.35	0.99	
Precision	0.63	1.00	0.24	0.88	
Recall	0.62	0.94	0.27	0.89	
F1-score	0.55	0.99	0.35	0.99	
Macro average	0.61	0.96	0.22	0.88	
Weighted average	0.49	0.99	0.38	0.98	

Fig. 11 Evaluation results of feature engineering bag-of-words experiment for the four algorithms.

Results of binary transformation using one-versus-one experiment

This binary transformation experiment was conducted using One-versus-One classifier. The results of this experiment show mostly similar results to binary transformation using One-versus-Rest classifier. The range of the acquiesced accuracy for this model lies between 72 and 81%. Table 9 and Fig. 12 provide the evaluation metrics for each algorithm in One-versus-one experiment.Table 9 One-vs-one experiment.

Algorithm	AdaBoost	CatBoost	RUSBoost	XGBoost	
Accuracy	0.81	0.78	0.72	0.80	
Precision	0.74	0.57	0.56	0.73	
Recall	0.61	0.49	0.61	0.58	
F1-score	0.81	0.78	0.72	0.80	
Macro average	0.64	0.50	0.55	0.62	
Weighted average	0.80	0.76	0.72	0.79	

Fig. 12 Evaluation results of binary transformation one-versus-one experiment for the four algorithms.

Results of binary transformation using one-versus-rest experiment

This experiment was conducted using a One-versus-Rest classifier. The range of the acquiesced accuracy for this model lies between 75 and 84%. Table 10 and Fig. 13 provide the evaluation metrics for each algorithm in the one-versus-rest experiment.Table 10 One-vs-rest experiment.

Algorithm	AdaBoost	CatBoost	RUSBoost	XGBoost	
Accuracy	0.83	0.79	0.75	0.84	
Precision	0.77	0.64	0.46	0.72	
Recall	0.67	0.54	0.57	0.65	
F1-score	0.83	0.79	0.75	0.84	
Macro avg	0.69	0.56	0.44	0.66	
Weighted avg	0.82	0.78	0.60	0.84	

Fig. 13 Evaluation results of binary transformation one-versus-rest experiment for the four algorithms.

Summary of results and discussion

In this part, our goal is to evaluate the experiments conducted in our research, with the aim of finding the best ensemble machine learning classifier combined with the best optimization technique. The choice of the most appropriate metric to evaluate a ML model depends on various factors such as the problem type and the nature of the data. While there are many metrics to choose from, classification accuracy is commonly used, particularly for binary classification problems. Another useful metric is the F1 Score, which represents the harmonic mean of precision and recall, and is often preferred when balancing precision and recall is important. It is particularly suitable when the class distribution is imbalanced as in our case. Hence, the F1 Score and accuracy metrics are reported in this section for evaluation purposes.AdaBoost algorithm

Figure 14 shows the F1-Score values of the experiment for the AdaBoost algorithm. The best performance for this algorithm has been achieved in the feature engineering TF-IDF experiment, where the F1-Score and accuracy equal 95% as shown in Table 7. Good accuracy has been achieved in binary transformation OvO and OvR experiments, with accuracy ranging from 72–81%, and 75–84% as shown in Tables 9 and 10, respectively. However, it shows bad performance in the first two experiments: non-hyperparameter and optimal hyperparameter.Fig. 14 F1-score: AdaBoost.

CatBoost algorithm

Figure 15 shows the F1-Score values of the experiment for the CatBoost algorithm. The best performance for this algorithm has been achieved in the feature engineering TF-IDF and bag-of-words experiments, where both the F1-Score and accuracy equal 99% for the two experiments as shown in Tables 7 and 8. The accuracy values of the CatBoost algorithm with optimal hyperparameter experiment as shown in Table 4 range from 63 to 81%. The accuracy values that have been achieved in OvO and OvR experiments range from 49–78%, and 45–79% as shown in Tables 9 and 10, respectively.Fig. 15 F1-score: CatBoost algorithm.

RUSBoost algorithm

Regarding RUSBoost, Fig. 16 shows the F1-Score values of this algorithm. The best performance of F1-Score has been achieved in the binary transformation experiments OvO and OvR with a value of 72% and 75% respectively. Although the performance of RUSBoost in the other experiments decreased to 16% for the optimal hyperparameter. For the accuracy metric, the algorithm shows better performance in the binary transformation OvO experiment with values between 55–72%, and 44–75% for the OvR as shown in Tables 4, 9, and 10, respectively.Fig. 16 F1-Score: RUSBoost algorithm.

XGBoost algorithm

The F1-score for the XGBoost algorithm is shown in Fig. 17, which has very good performance and outperformed other classifiers in all experiments. Accuracy of 100% has been achieved in the feature engineering TF-IDF experiment, and it’s similar to the result in the bag-of-words, where accuracy reaches up to 99%. In addition, it shows good accuracy for the other experiments with the values of 80%, 80%, 84% for optimal hyperparameter, OvO, and OvR, respectively as shown in Tables 6, 9, and 10.Fig. 17 F1-score: XGBoost algorithm.

In summary, The XGBoost and CatBoost algorithms outperform the rest of the classifiers in our study for both TF-IDF and bag-of-words. Regarding the optimization techniques, the feature engineering TF-IDF shows the best performance for all classifiers except for the RUSBoost algorithm. Finally, the best model in our work is XGBoost algorithm with TF-IDF categorizer which provides a performance value of 100% for accuracy, precision, recall, F1Score, Macro Avg, and Weighted. Table 11 summarizes the results of the twenty-four experiments.Table 11 Summary results of the conducted experiments.

Model	Metrics	(4 algorithms × 6 setups)	
Non hyper-parameter	Optimal hyper-parameter	Feature Eng. TF-IDF	Feature Eng. BOW	Binary Trans. OvO	Binary Trans. OvR	
XGBoost model	Accuracy	0.83	0.80	1.00	0.99	0.80	0.84	
Precision	0.75	0.73	1.00	0.88	0.73	0.72	
Recall	0.63	0.62	1.00	0.89	0.58	0.65	
F1-score	0.83	0.80	1.00	0.99	0.80	0.84	
Macro avg	0.66	0.64	1.00	0.88	0.62	0.66	
Weighted avg	0.82	0.80	1.00	0.98	0.79	0.84	
CatBoost model	Accuracy	0.81	0.81	0.99	0.99	0.78	0.79	
Precision	0.79	0.74	1.00	1.00	0.57	0.64	
Recall	0.62	0.63	0.94	0.94	0.49	0.54	
F1-score	0.81	0.81	0.99	0.99	0.78	0.79	
Macro avg	0.67	0.66	0.96	0.96	0.50	0.56	
Weighted avg	0.80	0.81	0.99	0.99	0.76	0.78	
AdaBoost model	Accuracy	0.37	0.37	0.95	0.55	0.81	0.83	
Precision	0.18	0.20	0.87	0.63	0.74	0.77	
Recall	0.23	0.24	0.83	0.62	0.61	0.67	
F1-score	0.37	0.37	0.95	0.55	0.81	0.83	
Macro avg	0.16	0.16	0.84	0.61	0.64	0.69	
Weighted avg	0.23	0.25	0.94	0.49	0.80	0.82	
RUSBoost model	Accuracy	0.26	0.16	0.58	0.35	0.72	0.75	
Precision	0.28	0.13	0.46	0.24	0.56	0.46	
Recall	0.19	0.11	0.38	0.27	0.61	0.57	
F1-score	0.26	0.16	0.58	0.35	0.72	0.75	
Macro avg	0.19	0.03	0.39	0.22	0.55	0.44	
Weighted avg	0.30	0.05	0.57	0.38	0.72	0.60	

Comparison with previous work

Several studies have concentrated on identifying and detecting refactoring activities throughout the software life cycle. For example, the study of39 which classifies refactoring documentation based on its type and used the same dataset as the one used in our study achieved 89.60% and 76.10% for the precision and recall metrics, respectively. As shown in Table 12, our results demonstrate a significant improvement in accuracy and F1-score of our work compared to previous studies.Table 12 Results comparison with previous work.

Study	Method	ML algorithms used	Classification type	Performance metrics	
Marmolejos et al.27	Feature extraction from commit messages	Bayes's point machine, averaged perceptron, logistic regression, boosted decision tree, neural network	Binary (refactoring or not)	Accuracy and F1-score up to 96%	
AlOmar et al.5	NLP preprocessing and N-gram feature extraction	9 ML algorithms	Multiclass (6 categories at method level)	Accuracy up to 93% for some classes, 45% for others	
Nyamawe28	Two classification methods	Not specified	Binary (refactoring or not), Multilabel (refactoring types)	F1-score up to 87.56% (binary), Accuracy up to 80.86% (multilabel)	
AlOmar et al.29	Classification of 800 Java projects	Not specified	Multiclass (internal quality, external quality, code smell resolution)	Not specified	
Aniche30	Detection of refactoring types	6 ML algorithms, RandomForest	Multiclass	Accuracy up to 90%	
Sheneamer31	Automatic detection based on code clones	Not specified	Multiclass	Accuracy up to 99.5%	
Krasniqi and Cleland-Huang32	Two-phase crime classification	SVM	Binary (CMR or non-CMR), Multiclass (12 refactoring types)	F1-score up to 84% (binary), 71% (multiclass)	
Nyamawe et al.33	Detection in 43 open-source Java projects	Not specified	Binary and multilabel	F1-score 74.63%	
Gharbi et al.34	Multi-label classification using binary relevance	Logistic regression	Multilabel (3 maintenance activities)	F1-score 45.79%	
Fontana et al.35	Detection of code smells	J48, JRip, Random Forest, Naïve Bayes, SMO, LibSVM	Multiclass	Accuracy ~ 96%	
Levin and Yehudai37	Manual labeling and classification	GBM, J48, Random Forest	Multiclass (preventive, corrective, adaptive)	Accuracy up to 76%	
Rodriguez et al.38	Clustering approach for refactoring suggestions	COBWEB clustering algorithm	Clustering	Not specified	
Atwi et al.39	Automated approach (PYREF)	Manual	Multiclass	Precision 89.60%, Recall 76.10%	
Proposed work	Feature extraction from commit messages

NLP preprocessing

Detection of refactoring types

	AdaBoost, CatBoost, RUSBoost, XGBoost	Detection of 10 refactoring categories	Highest precision: 100%

Highest Recall: 100%

	

The intersection between our work and the studies mentioned in Table 12 is that we present an approach based on machine learning where the refactoring type is detected by a machine learning algorithm without manual intervention. Additionally, the performance of the different ensemble learning models was compared. It is important to note that unlike other related studies, the present work aims to identify whether refactoring has occurred or not and demonstrate the distinct executed refactoring types. We aimed in our study to identify refactoring types within 10 categories: Adding Parameter, Renaming Method, Moving Method, Changing/Renaming Parameter, Removing Parameter, Extracting Method, Pulling Up Method, Inlining Method, and Pushing Down Method. Besides increasing the accuracy of refactoring identification, this classification level also provides additional understanding of the types of refactorings that have been carried out, thus helping to improve software maintenance and quality methods.

Finally, because of the high accuracy and F1-score in the discovery of refactoring types using machine learning algorithms in the proposed approach, it distinguishes it from other studies. The use of enhanced ensemble learning models results in improved precision and recall by increasing the accuracy of refactoring activities’ identification. These enhancements in performance measures show the efficiency and stability of the proposed method in discriminating refactoring types; thus, it gains substantial ground in the existing theories and applications of software engineering and aids development and research professionals considerably.

Conclusion and future work

This paper introduced an ensemble approach for detecting refactoring types. A refactoring documentation dataset has been used, which includes 573 commit documentations for the following projects: DIT6, TEXAR7, FFMPEGPYTHON8, and nine refactoring types. Four ensemble boosting ML algorithms have been used: the AdaBoost, CatBoost, RUSBoost, and XGBoost algorithms. Low accuracy has been achieved due to data imbalances and multi-class data issues. Three enhancement experiments have been performed to achieve acceptable accuracy. The experiments are hyperparameter optimization using GridSearchCV, feature engineering using TF-IDF or a Bag-of-Words, and binary transformation using one-vs-one and one-vs-rest classifiers. Each experiment has been reflected in a different way on each classifier. However, the best model that we achieved is the XGBoost classifier with the TF-DF vectorizer.

Limitations of this work are summarized as follows: First, the generalization of our findings. The study only utilized one dataset in a certain format, and the performance of the models might vary if the input dataset is different. Second, this study classifies refactoring into nine types, developers may require more types of refactoring in their work. Finally, this study used only two hyperparameters to find the optimal one. Several hyperparameters might be used that provide better performance.

In the future, we aim to use different refactoring documentation datasets to see how this approach performs with different datasets. Also, we aim to explore deep-learning approaches and more ensemble machine-learning algorithms, such as LogitBoost. In addition, we aim to explore other enhancement techniques to solve imbalance and multiclass data issues.

Replication

To replicate the refactoring detection using Python, the code is accessed at (https://github.com/nasbaih/RefactoringCommits-Type-Detection). The dataset used in this research is available at (https://github.com/PyRef/PyRef)39.

Author contributions

Dimah Al-Fraihat: Methodology, software, formal analysis, visualization, writing-original draft preparation, writing-reviewing and editing, supervision, administration. Yousef Sharrab: Writing-reviewing and editing, visualization, validation, investigation. Abdel-Rahman Al-Ghuwairi: Conceptualization. Nour Sbaih: Software, formal analysis, visualization. Ayman Qahmash: Writing-reviewing and editing, and funding.

Funding

This research was funded by the Deanship of Scientific Research at King Khalid University, grant number RGP.1/209/43.

Data availability

The data presented in this study are available and can be accessed at: (https://github.com/nasbaih/Refactoring-Commits-Type-Detection). The dataset used in this research is available at (https://github.com/PyRef/PyRef).

Competing interests

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Peruma A Simmons S AlOmar EA Newman CD Mkaouer MW Ouni A How do I refactor this? An empirical study on refactoring trends and topics in stack overflow Empir. Softw. Eng. 2022 27 1 11 10.1007/s10664-021-10045-x
Peruma, A. et al. How do I refactor this? An empirical study on refactoring trends and topics in stack overflow. Empir. Softw. Eng. 27(1), 11 (2022).10.1007/s10664-021-10045-x
2. Das, A. K., Yadav, S., Dhal, S. Detecting code smells using deep learning. In TENCON 2019–2019 IEEE Region 10 Conference (TENCON) 2081–2086 (IEEE, 2019).
3. Hammad M Labadi A Automatic detection of bad smells from code changes Int. Rev. Comput. Softw. 2016 11 11 1016 1027
Hammad, M. & Labadi, A. Automatic detection of bad smells from code changes. Int. Rev. Comput. Softw. 11(11), 1016–1027 (2016).
4. Al-Fraihat, D., Sharrab, Y., Al-Ghuwairi, A. R., Alshishani, H., & Algarni, A. Hyperparameter optimization for software bug prediction using ensemble learning. IEEE Access. (2024).
5. AlOmar EA Liu J Addo K Mkaouer MW Newman C Ouni A Yu Z On the documentation of refactoring types Autom. Softw. Eng. 2022 29 1 40 10.1007/s10515-021-00314-w
AlOmar, E. A. et al. On the documentation of refactoring types. Autom. Softw. Eng. 29, 1–40 (2022).10.1007/s10515-021-00314-w
6. Di Penta, M., Bavota, G., Zampetti, F. On the relationship between refactoring actions and bugs: A differentiated replication. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 556–567 (2020).
7. Al-Fraihat D Sharrab Y Al-Ghuwairi AR AlElaimat M Alzaidi M Detecting and resolving feature envy through automated machine learning and move method refactoring Int. J. Electr. Comput. Eng. IJECE. 2024 14 2 2330 2343
Al-Fraihat, D., Sharrab, Y., Al-Ghuwairi, A. R., AlElaimat, M. & Alzaidi, M. Detecting and resolving feature envy through automated machine learning and move method refactoring. Int. J. Electr. Comput. Eng. IJECE. 14(2), 2330–2343 (2024).
8. Rebai S Kessentini M Alizadeh V Sghaier OB Kazman R Recommending refactorings via commit message analysis Inf. Softw. Technol. 2020 126 106332 10.1016/j.infsof.2020.106332
Rebai, S., Kessentini, M., Alizadeh, V., Sghaier, O. B. & Kazman, R. Recommending refactorings via commit message analysis. Inf. Softw. Technol. 126, 106332 (2020).10.1016/j.infsof.2020.106332
9. Tian, Y., Zhang, Y., Stol, K.-J., Jiang, L., Liu, H. What makes a good commit message? In Proceedings of the 44th International Conference on Software Engineering, 2389–2401 (2022).
10. Karakati CB Thirumaaran S Software code refactoring based on deep neural network-based fitness function Concurr. Comput. Pract. Exp. 2023 35 4 e7531 10.1002/cpe.7531
Karakati, C. B. & Thirumaaran, S. Software code refactoring based on deep neural network-based fitness function. Concurr. Comput. Pract. Exp. 35(4), e7531 (2023).10.1002/cpe.7531
11. Mens T Tourwe T A survey of software refactoring IEEE Trans. Softw. Eng. 2004 30 2 126 139 10.1109/TSE.2004.1265817
Mens, T. & Tourwe, T. A survey of software refactoring. IEEE Trans. Softw. Eng. 30(2), 126–139 (2004).10.1109/TSE.2004.1265817
12. Guo Y Wang X Xiao P Xu X An ensemble learning framework for convolutional neural network based on multiple classifiers Soft Comput. 2020 24 3727 3735 10.1007/s00500-019-04141-w
Guo, Y., Wang, X., Xiao, P. & Xu, X. An ensemble learning framework for convolutional neural network based on multiple classifiers. Soft Comput. 24, 3727–3735 (2020).10.1007/s00500-019-04141-w
13. Freund, Y., Schapire, R. E. et al. Experiments with a new boosting algorithm. In icml, vol. 96, 148–156 (Citeseer, 1996).
14. Yue, R., Gao, Z., Meng, N., Xiong, Y., Wang, X., Morgenthaler, J. D. Automatic clone recommendation for refactoring based on the present and the past. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME) 115–126 (IEEE, 2018).
15. Panigrahi, R., Kuanar, S. K., Kumar, L. Empirical study on method-level refactoring using machine learning. In Next Generation of Internet of Things: Proceedings of ICNGIoT 2022 663–673 (Springer, 2022).
16. Alkharabsheh K Alawadi S Kebande VR Crespo Y Fernandez-Delgado M Taboada JA A comparison of machine learning´ algorithms on design smell detection using balanced and imbalanced dataset: A study of god class Inf. Softw. Technol. 2022 143 106736 10.1016/j.infsof.2021.106736
Alkharabsheh, K. et al. A comparison of machine learning´ algorithms on design smell detection using balanced and imbalanced dataset: A study of god class. Inf. Softw. Technol. 143, 106736 (2022).10.1016/j.infsof.2021.106736
17. Al-Fraihat, D., Sharrab, Y., Alzyoud, F., Qahmash, A., Tarawneh, M., & Maaita, A. Speech recognition utilizing deep learning: A systematic review of the latest developments. Hum.-Centric Comput. Inf. Sci. 14 (2024).
18. Kumar, L., Satapathy, S. M., Murthy, L. B. Method level refactoring prediction on five open source java projects using machine learning techniques. In Proceedings of the 12th Innovations on Software Engineering Conference (Formerly Known as India Software Engineering Conference) 1–10 (2019).
19. Al-Ghuwairi AR Al-Fraihat D Sharrab Y Alrashidi H Almujally N Kittaneh A Ali A Visualizing software refactoring using radar charts Sci. Rep. 2023 13 1 19530 10.1038/s41598-023-44281-6 37945685
Al-Ghuwairi, A. R. et al. Visualizing software refactoring using radar charts. Sci. Rep. 13(1), 19530 (2023).37945685 10.1038/s41598-023-44281-6
20. Jain S Saha A Improving performance with hybrid feature selection and ensemble machine learning techniques for code smell detection Sci. Comput. Program. 2021 212 102713 10.1016/j.scico.2021.102713
Jain, S. & Saha, A. Improving performance with hybrid feature selection and ensemble machine learning techniques for code smell detection. Sci. Comput. Program. 212, 102713 (2021).10.1016/j.scico.2021.102713
21. Khanna M Singh LK Thawkar S Goyal M Deep learning based computer-aided automatic prediction and grading system for diabetic retinopathy Multimed. Tools Appl. 2023 82 25 39255 39302 10.1007/s11042-023-14970-5
Khanna, M., Singh, L. K., Thawkar, S. & Goyal, M. Deep learning based computer-aided automatic prediction and grading system for diabetic retinopathy. Multimed. Tools Appl. 82(25), 39255–39302 (2023).10.1007/s11042-023-14970-5
22. Khanna M Singh LK Thawkar S Goyal M PlaNet: a robust deep convolutional neural network model for plant leaves disease recognition Multimedia Tools and Applications 2024 83 2 4465 4517 10.1007/s11042-023-15809-9
Khanna, M., Singh, L. K., Thawkar, S. & Goyal, M. PlaNet: a robust deep convolutional neural network model for plant leaves disease recognition. Multimedia Tools and Applications 83(2), 4465–4517 (2024).10.1007/s11042-023-15809-9
23. Yang L Shami A On hyperparameter optimization of machine learning algorithms: Theory and practice Neurocomputing 2020 415 295 316 10.1016/j.neucom.2020.07.061
Yang, L. & Shami, A. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing 415, 295–316 (2020).10.1016/j.neucom.2020.07.061
24. Dong, G., Liu, H. Feature Engineering for Machine Learning and Data Analytics (CRC Press, 2018).
25. Liu, Q., Wang, J., Zhang, D., Yang, Y., Wang, N. Text features extraction based on TF-IDF associating semantic. In 2018 IEEE 4th International Conference on Computer and Communications (ICCC) 2338–2343 (IEEE, 2018).
26. Scott, S., Matwin, S. Feature engineering for text classification. In ICML, vol. 99, 379–388 (1999).
27. Xue H Sun S Venkataramani G Lan T Machine learning-based analysis of program binaries: A comprehensive study IEEE Access 2019 7 65889 65912 10.1109/ACCESS.2019.2917668
Xue, H., Sun, S., Venkataramani, G. & Lan, T. Machine learning-based analysis of program binaries: A comprehensive study. IEEE Access 7, 65889–65912 (2019).10.1109/ACCESS.2019.2917668
28. Marmolejos, L., AlOmar, E. A., Mkaouer, M. W., Newman, C., Ouni, A. On the use of textual feature extraction techniques to support the automated detection of refactoring documentation. Innov. Syst. Softw. Eng. 1–17 (2021).
29. Nyamawe AS Mining commit messages to enhance software refactorings recommendation: A machine learning approach Mach. Learn. Appl. 2022 9 100316
Nyamawe, A. S. Mining commit messages to enhance software refactorings recommendation: A machine learning approach. Mach. Learn. Appl. 9, 100316 (2022).
30. AlOmar EA Peruma A Mkaouer MW Newman C Ouni A Kessentini M How we refactor and how we document it? On the use of supervised machine learning algorithms to classify refactoring documentation Expert Syst. Appl. 2021 167 114176 10.1016/j.eswa.2020.114176
AlOmar, E. A. et al. How we refactor and how we document it? On the use of supervised machine learning algorithms to classify refactoring documentation. Expert Syst. Appl. 167, 114176 (2021).10.1016/j.eswa.2020.114176
31. Aniche M Maziero E Durelli R Durelli VH The effectiveness of supervised machine learning algorithms in predicting software refactoring IEEE Trans. Softw. Eng. 2020 48 4 1432 1450 10.1109/TSE.2020.3021736
Aniche, M., Maziero, E., Durelli, R. & Durelli, V. H. The effectiveness of supervised machine learning algorithms in predicting software refactoring. IEEE Trans. Softw. Eng. 48(4), 1432–1450 (2020).10.1109/TSE.2020.3021736
32. Sheneamer AM An automatic advisor for refactoring software clones based on machine learning IEEE Access 2020 8 124978 124988 10.1109/ACCESS.2020.3006178
Sheneamer, A. M. An automatic advisor for refactoring software clones based on machine learning. IEEE Access 8, 124978–124988 (2020).10.1109/ACCESS.2020.3006178
33. Krasniqi, R., Cleland-Huang, J. Enhancing source code refactoring detection with explanations from commit messages. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), 512–516 (IEEE, 2020).
34. Nyamawe, A. S., Liu, H., Niu, N., Umer, Q., Niu, Z. Automated recommendation of software refactorings based on feature requests. In 2019 IEEE 27th International Requirements Engineering Conference (RE) 187–198 (IEEE, 2019).
35. Gharbi, S., Mkaouer, M. W., Jenhani, I., Messaoud, M. B. On the classification of software change messages using multi-label active learning. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing 1760–1767 (2019).
36. Arcelli Fontana F Mantyla MV Zanoni M Marino A Comparing and experimenting machine learning techniques for code smell detection Empir. Softw. Eng. 2016 21 1143 1191 10.1007/s10664-015-9378-4
Arcelli Fontana, F., Mantyla, M. V., Zanoni, M. & Marino, A. Comparing and experimenting machine learning techniques for code smell detection. Empir. Softw. Eng. 21, 1143–1191 (2016).10.1007/s10664-015-9378-4
37. Di Nucci, D., Palomba, F., Tamburri, D. A., Serebrenik, A., De Lucia, A. Detecting code smells using machine learning techniques: Are we there yet? In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER) 612–621 (IEEE, 2018).
38. Levin, S., Yehudai, A. Boosting automatic commit classification into maintenance activities by utilizing source code changes. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering 97–106 (2017).
39. Rodriguez, G., Soria, A., Teyseyre, A., Berdun, L., Campo, M. Unsupervised learning for detecting refactoring opportunities in service-oriented applications. In Database and Expert Systems Applications: 27th International Conference, DEXA 2016, Porto, Portugal, September 5–8, 2016, Proceedings, Part II 27 335–342 (Springer, 2016).
40. Atwi, H., Lin, B., Tsantalis, N., Kashiwa, Y., Kamei, Y., Ubayashi, N., Bavota, G., Lanza, M. Pyref: Refactoring detection in python projects. In 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM) 136–141 (IEEE, 2021).
41. Hattori, L. P., Lanza, M. On the nature of commits. In 2008 23rd IEEE/ACM International Conference on Automated Software Engineering-Workshops 63–71 (IEEE, 2008).
42. Borg, R., Kropp, M. Automated acceptance test refactoring. In Proceedings of the 4th Workshop on Refactoring Tools 15–21 (2011).
43. Kashiwabara, Y., Onizuka, Y., Ishio, T., Hayase, Y., Yamamoto, T., Inoue, K. Recommending verbs for rename method using association rule mining. In 2014 Software Evolution Week-IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMRWCRE) 323–327 (IEEE, 2014).
44. Peruma, A., Mkaouer, M. W., Decker, M. J., Newman, C. D. An empirical investigation of how and why developers rename identifiers. In Proceedings of the 2nd International Workshop on Refactoring 26–33 (2018).
45. Kataoka, Y., Ernst, M. D., Griswold, W. G., Notkin, D. Automated support for program refactoring using invariants. In Proceedings IEEE International Conference on Software Maintenance. ICSM 2001 736–743 (IEEE, 2001).
46. Oliveira J Gheyi R Mongiovi M Soares G Ribeiro M Garcia A Revisiting the refactoring mechanics Inf. Softw. Technol. 2019 110 136 138 10.1016/j.infsof.2019.03.002
Oliveira, J. et al. Revisiting the refactoring mechanics. Inf. Softw. Technol. 110, 136–138 (2019).10.1016/j.infsof.2019.03.002
47. Tensorflow-keras-library. (2023) https://www.tensorflow.org/apidocs/python/tf/keras (accessed 22 Dec 2023).
48. Plisson, J., Lavrac, N., Mladenic, D. et al. A rule based approach to word lemmatization. In Proceedings of IS, vol. 3, 83–86 (2004).
49. Bani-Salameh, H., Sallam, M., Al shboul, B. A deep-learning-based bug priority prediction using rnn-lstm neural networks. e-Inf. Softw. Eng. J. 15(1) (2021).
50. Hu Y Zhang X-Q Xu L He FX Tian Z She W Liu W Harmonic loss function for sensor-based human activity recognition based on lstm recurrent neural networks IEEE Access 2020 8 135617 135627 10.1109/ACCESS.2020.3003162
Hu, Y. et al. Harmonic loss function for sensor-based human activity recognition based on lstm recurrent neural networks. IEEE Access 8, 135617–135627 (2020).10.1109/ACCESS.2020.3003162
51. Sindjoung, M. L. F., Minet, P. Estimating and predicting link quality in wireless iot networks. Ann. Telecommun. 1–13 (2021).
52. Singh J Singh J Assessment of supervised machine learning algorithms using dynamic api calls for malware detection Int. J. Comput. Appl. 2022 44 3 270 277
Singh, J. & Singh, J. Assessment of supervised machine learning algorithms using dynamic api calls for malware detection. Int. J. Comput. Appl. 44(3), 270–277 (2022).
53. Ranjan, G., Verma, A. K., Radhika, S. K-nearest neighbors and grid search cv based real time fault monitoring system for industries. In 2019 IEEE 5th International Conference for Convergence in Technology (I2CT) 1–5 (IEEE, 2019).
54. Sklearn-library. (2023). https://pypi.org/project/sklearn/ (accessed 17 Dec 2023).
55. Dias Canedo E Cordeiro Mendes B Software requirements classification using machine learning algorithms Entropy. 2020 22 9 1057 10.3390/e22091057 33286826
Dias Canedo, E. & Cordeiro Mendes, B. Software requirements classification using machine learning algorithms. Entropy. 22(9), 1057 (2020).33286826 10.3390/e22091057
56. Ali, K., Alzaidi, M., Al-Fraihat, D., & Elamir, A. M. Artificial intelligence: Benefits, application, ethical issues, and organizational responses. In Intelligent Sustainable Systems: Selected Papers of WorldS4 2022, vol. 1, 685–702 (Springer Nature Singapore, 2023).
57. Lorena AC De Carvalho AC Gama JM A review on the combination of binary classifiers in multiclass problems Artif. Intell. Rev. 2008 30 19 37 10.1007/s10462-009-9114-9
Lorena, A. C., De Carvalho, A. C. & Gama, J. M. A review on the combination of binary classifiers in multiclass problems. Artif. Intell. Rev. 30, 19–37 (2008).10.1007/s10462-009-9114-9
58. Al-Obeidallah, M. G., Al-Fraihat, D. G., Khasawneh, A. M., Saleh, A. M., & Addous, H. Empirical investigation of the impact of the adapter design pattern on software maintainability. In 2021 International Conference on Information Technology (ICIT) 206–211 (IEEE, 2021).
