Related works

One of the features provided by most IMS is to assign a category or a set of labels to the generated issues with the aim, at least in theory, to facilitate their management and retrieval. Several authors have studied the use of labels to categorize issues in an IMS. In ⁴, the authors analyzed a population of more than three million GitHub projects and gave some insights on how labels are used in them. Their results reveal that, even if the label mechanism is scarcely used, using labels favors the resolution of issues. They also conclude that not all projects use labels similarly (e.g., for some, labels are only a way to prioritize the project, while others use them to signal their temporal evolution as they move along in the development workflow).

In a study conducted on closed issue reports of three open-source software systems from Jira, it has been observed that the label given to the issue reports about bugs or improvement is incorrect ⁵. The authors manually classified more than 7000 closed issue reports from five popular open-source software systems to analyze the accuracy of already labeled reports. Their findings state that 33.8% of closed issue reports were misclassified.

The authors in ⁶ manually classified a dataset and applied machine learning algorithms for bug classification. In ⁽⁷, an automated approach is proposed to label an issue either as a bug or other request based on fuzzy set theory. The labeling of bug reports is done in three phases. First, text from the bug reports is preprocessed. Second, the Fuzzy technique is applied, and third, the labeling is done using scores obtained after fuzzification. In ⁸, the authors selected seven projects in GitHub and built classification models based on issue information, text descriptions, and comments to improve the maintenance tasks for development teams. Text information was preprocessed with text data mining techniques and information retrieval. Then, they evaluated the performance of classifiers with several metrics. They conclude that very suitable classifiers may be obtained to label the issues or suggest the most suitable candidate labels.

These contributions employed datasets obtained from repositories of IMS configured for bug tracking and not for project management. For that reason, the focus of these works has been on the correct classification/labeling of defects or bugs. However, our work employs datasets obtained from IMS repositories used for project management and software development, and we focused on the identification of issues related to requirements definition, such as "user stories".

In the last years, a research trend has emerged regarding applying computational linguistic techniques to user stories to solve classic challenges in requirements engineering, such as the formulation of high-quality requirements or the creation of better models of system functionalities ⁹. A research line is the extraction of conceptual models from natural language (NL) requirements, which can help to identify dependencies, redundancies, and conflicts between requirements from lengthy textual specifications. To extract meaningful models from requirements expressed in NL, researchers have been proposing heuristic rules for the identification of entities and relationships whenever the text matches particular patterns of the given language (usually English). For example, in ⁹, is proposed an automated approach based on natural language processing that extracts conceptual models from user story requirements. In another work ¹⁰, the authors proposed an approach to generate i* models from user stories. In ¹¹, contributions are made toward mapping user stories and use case models. Also, in ¹², user stories are used to extract quality attributes for early architecture decision-making. A common denominator of all these proposals is that they require user stories as input, so mislabeled user stories harm the results of such studies. Consequently, to anticipate better results from these user story studies, an approach that correctly identifies subjects as either "user stories" or "non-user stories" is required.

Background

In this section, the theoretical concepts on which this work is based are exposed. We describe the concepts and models used in this paper, such as User Stories, Recurrent Neural Networks, Bidirectional Long Short-Term Memory Recurrent Neural Networks, and Natural Language Processing with Neural Networks, among others.

User stories

Outside the world of software, a user story could be referred to as a customer's testimonial or narrative; however, it has a whole different meaning for software professionals. In terms of software development, a user story is a short description of something or a piece of software it is supposed to do, told from the perspective of the person who desires the new feature. Although going back to its beginnings, user stories were proposed as unstructured text but with some size restrictions ¹, nowadays, it follows a compact template for writing them. The template captures who it is, its expectations of the system function, and, optionally, why it is significant ⁽¹³. Although many different templates exist, 70% of practitioners use the template: "As a (type of user), I want (goal), [so that (some reason)]" ⁽¹. Next, two examples of user stories using such a template are introduced.

• Example 1: As a visitor, I want to purchase an event ticket.

• Example 2: As an event organizer, I want to search for new events by favorited organizers, So that I know of events first.

Natural language processing with neural networks

Natural Language Processing (NPL) is a subfield of linguistic, computer, information engineering, and artificial intelligence sciences dedicated to interacting with computer equipment and human natural language, particularly how computer programs process and analyze large amounts of information. The problems often addressed with these techniques are speech recognition, understanding natural language such as sentiment analysis, text generation, automatic text summarization, and automatic entity recognition¹⁴. Although there exist several natural language processing techniques, in recent years, there has been a significant boom in the use of Deep Learning models ¹⁴ because of their ability to capture the syntactic and semantic information of words in large unlabeled bodies of text. Word vectors (word embeddings) are a standard component found in current NLP system architectures ¹⁴. Word embeddings are vectors of real numbers representing terms correlating relative similarities with semantic similarities ¹⁵, generally learned by neural networks. They can represent the context of the word and can provide information about relations with other words. Hence, the meaning or semantic context of words can be predicted accurately as they can capture syntactic and semantic information about the words ¹⁶. Following this trend, there are analyzed and used popular models at the moment of this work.

Recurrent Neural Network (RNN) architectures have become a typical and famous neural network model because of their capabilities to process sequential inputs and learn its dependencies ¹⁷, proving to be very helpful in NLP tasks. An RNN is a neural network where the connections between neurons form a directed graph, making a temporal sequence through X _t time steps, feeding each hidden state H _t to the next time step, as shown in Figure 3. The network thus has a dynamic temporal behavior. Unlike common networks, RNNs can use an internal state (memory state) to process sequences of inputs. However, they have problems with long-term dependencies due to gradient vanishing ¹⁷.

Figure 3 Recurrent Neural Network. 

Otherwise, long short-term memory (LSTM) is a recurrent neural network architecture type that avoids the problem of gradient vanishing. LSTM is augmented by recurrent “forgetting” gates, preventing the backward propagation error from vanishing or exploding. In this type of network errors can go backward through a virtually unlimited number of layers unfolded in space. As shown in Figure 4, the internal memory cell C _t is controlled by a set of gate networks: a forget gate network f _t an input gate network i _t , and an output gate network o _t . The forget gate network controls how much information of the internal cell C _t should be passed into the next time step. The input gate network is used to scale the input block u _t to the internal cell. Consequently, LSTM can learn tasks that require memories of events that occurred thousands of times in previous training steps, thus making it capable of handling long-term dependencies ¹⁷.

Figure 4 Schematic of the LSTM. 

On the other hand, Bi-directional Recurrent Neuronal Networks (BRNN) have a specific structure. The state neurons of a regular RNN are split into a part that is responsible for the positive time direction (forward states) and a part for the negative time direction (backward states), as shown in Figure 5. These outputs of two types of states are not necessarily connected to inputs in the opposite states ¹⁸. Using time directions in the same network, input information in the past (t-1 in Figure 5) and the future (t+1 in Figure 5) of the currently evaluated time frame (t) can be used to minimize the objective function without the need for delays, unlike common RNN that require these "delays" to include future information. Using the LSTM and BRNN models, the model can handle long-term dependencies and analyze the whole sentence forward and backward ¹⁹.

Figure 5 General structure of the bidirectional recurrent neural network (BRNN) shown unfolded in time for three-time steps. 

Commonly, nowadays, different NLP tasks entail a great effort in terms of time and computing power consumption, so as an alternative to creating a model from scratch or too general, the transfer learning technology has emerged ²⁰. Transfer Learning (TL) is a machine learning method with the perspective of providing a better and faster solution with less effort for collecting the needed training information and reusing it in another similar model ²⁰. In ²⁰, it is defined as: "Given a Ds domain and a source Ts learning task, and a Dt domain and target Tt learning task, the TL aims to enhance the learning of the target predictive function f(x) in Dt using the knowledge in Ds and Ts, where Ds ≠ Dt, or Ts ≠Tt." Word embeddings are a good example of transfer learning since neural networks generally learn them in a domain for a learning task, and these learned word embeddings can be applied in a different domain for other learning tasks. Hence, those vectors of real numbers are transferred from one model to another.

Word representations, such as Word Embeddings, are a crucial component in many neural language models ²¹. ELMo (Embeddings from Language Models) incorporates a form of deep word representation based on a feature-based approach, where each token is assigned a representation that is a function of the entire input sequence ²¹. The vectors derived from a trained LSTM network with a pair of linguistic models are used in an extended text corpus. These representations are a function of all the layers of a Bidirectional Linguistic Model (biLM) ²¹. ELMo looks at the entire sentence before assigning each word in its embedding. It uses a bi-directional LSTM trained on a specific task to create contextual word embedding. The ELMo LSTM, once trained on a massive dataset, could be used as a constituent in other NLP models aimed at language modeling. In ⁽²², an implementation of a module with this architecture and an application trained in 1 billion words is presented. This module returns as output a set of fixed embeddings for each LSTM layer, the learned aggregation composed of 3 layers, and a mean-pooled vector representation of the input.

There are two strategies for applying pre-training in linguistic models: the characteristics-based approach and the parameter adjustment approach ⁽²³. Feature-based models such as ELMo ²¹ use architectures that include pre-trained representations as additional features. On the other hand, models that use parameter resetting introduce parameters to specific tasks, trying to simplify and adjust all the pre-trained parameters. However, current techniques based on the parameter-matching approach use unidirectional linguistic models ²³. BERT (Bidirectional Encoder Representations from Transformers) ²³ alleviates this problem using a masked linguistic model. The linguistic model masks some of the input tokens and aims to predict the original id of the vocabulary by linking the contexts from the right and left; hence it is bidirectional.

BERT uses a masked language modeling objective to pre-train the transformer network on an extensive unlabeled data ²⁴. In ²⁵, it can be found an implementation and examples of the use of a module that fits this architecture trained in Wikipedia and BookCorpus. Assuming that the entries are pre-processed as required by this module implementation, it returns as output representations of each token in the input sequence and an entire grouped representation of the entry.

Attention mechanisms have become an integral part of sequential modeling in various tasks, allowing the modeling of dependencies regardless of the distance between input and output sequences. These mechanisms are generally used with some RNN²⁶. These models use the so-called attention functions, which are nothing more than a function that can be described as the mapping of a query and a set of identifier-value pairs to an output, where the query, the identifiers, and the values are all vectors. The output is calculated as the weighted sum of the values, where the weight of each value is calculated by a query compatibility function with the corresponding identifier ²⁶. In ²⁶, various types of attention functions, such as "Scaled Dot-Product Attention," "Multi-Head Attention," and "Self- Attention," are presented and explained.

A model called "Transformer" ²⁶ is completely based on the Self-Attention and Multi-Head Attention models. This model does not use alienated RNNs or convolutions; it follows an encoder-decoder architecture completely connected between its layers. That means that the encoder maps an input sequence of symbol representations to a continuous representation. Then, the decoder generates an output of the symbols for each element at a time ²⁷.

Data understanding and preparation

The models were trained by taking data from public sources containing real software development project problems ^{29) (30}. These sources contain positive examples of user stories (sentences in the format described previously) and negative examples (erroneous user stories or sentences with a similar syntaxis to user stories but with a different purpose). An algorithm was implemented to generate additional examples by splitting and mixing positive examples into random parts using the TensorFlow Tokenizer to obtain a more extensive data set suitable for testing the models. This implementation is available in ³¹. A manual classification work was performed to differentiate the examples to which each classification class belonged, thus introducing into the model an index of human error, given that there was no record of the previously classified data. The resulting dataset includes a total of 7997 positive and negative examples, of which 2618 are positive, as shown in Table 1, and the remainder are negative, as shown in Table 2.

Table 1 Samples of positive examples in the dataset. 

Table 2 Samples of negative examples in the dataset. 

Therefore, a binary classification problem is presented, where the issues classified as user stories belong to the positive class (1) and the rest to the negative class (0). The entire dataset obtained can be found in ³².

A BRNN-LSTM model for User Story issues classification

The first model proposed for User Stories classification is based on an architecture for a bidirectional LSTM neural network (Figure 6). The model has a maximum of Word Embeddings equal to the vocabulary length, with 300 dimensions each and 125 bidirectional LSTM layers. A dropout layer was used to prevent overfitting, and a sigmoid activation function in the output layer.

For the implementation of this model, Python 3 and TensorFlow 2.0.0-rc0 for GPUs were used. The implemented model is available in ³³.

Figure 6 The BRNN-LSTM model. 

An ELMo-based model for User Story issues classification

Furthermore, a custom Keras layer for TensorFlow, whose implementation was taken from ³⁵ and subsequently integrated and adapted to our model, was used to build the ELMo-based model using the ELMo2 module available through the Tensorflow Hub platform ³⁴. Besides, a dropout layer was added to the model to prevent overfitting and a sigmoid activation function. Figure 7 illustrates a general view of the sequential model using the ELMo module.

Figure 7 The proposed model using ELMo module. 

For this implementation, Tensor Flow 1.14 was used due to support and compatibility problems of the module with TensorFlow2.0 and the Tensor Flow-Hub library ³⁶. The model implementation is available in ³⁷.

A BERT-based model for User Story issues classification

The BERT module bert_uncased_L-12_H-768_A-12/1, available through the Tensorflow Hub platform ³⁴, which provides a simple way to share Tensorflow models, was used to implement the BERT-based model. We used Keras with Tensorflow backend to build our BERT-based model. Before Keras can use the core TensorFlow model, a customized Keras layer must be defined to render it in the appropriate format ³⁸ correctly.

As shown in Figure 8, after the inputs are preprocessed, the ids for the tokens and their respective masks are obtained, which fed the BERT layer. Finally, to avoid overfitting, a dropout layer is placed at the output of the BERT module, and subsequently, a sigmoid function is used.

Figure 8 The proposed model using BERT module. 

The implementation of this model used Tensor Flow 1.14 to avoid some compatibility issues with TensorFlow2.0 and the Tensor Flow-Hub library. The implemented model is available in ⁽³⁹.

Training and validating the models

For the BRNN-LSTM model, an Adam optimizer with a learning rate of 0.01 and a batch size of 35 was used, as shown in Figure 9 after 15 epochs of 23 seconds each, accuracies of 0.9579 and 0.9624 were achieved in validation and testing, respectively.

Figure 9 Training of the BRNN-LSTM model. 

In Figure 10, it is analyzed the performance graphs by accuracy and loss. As it can be observed, there exist some overfitting, which could lead to missing classifications.

Figure 10 Accuracy and Loss analysis of the BRNN-LSTM model. 

For validating the automatic generation algorithm, the same training and validation process was run for the BRNN-LSTM model using the original dataset. After that, the F1 score was 0.8796 against 0.9545 for the model trained using the enhanced dataset, as shown in Figure 11. As can be seen, the model using the enhanced dataset has a better score; hence, it was decided to continue the training and validation of the rest of the models using the enhanced data set only.

Figure 11 Training BRNN-LSTM model with original vs enhanced dataset. 

For the ELMo-based model, an SGD optimizer and a batch size of 35 were used, and after 34 epochs of 23 seconds each, an accuracy of 0.9607 in validation was obtained, as shown in Figure 12.

Figure 12 Training the ELMo-based model. 

An analysis of the performance of this model shows a better performance than the previous one without a relevant overfitting (Figure 13).

Figure 13 Accuracy and Loss analysis for the ELMo-based model. 

On the other hand, for the BERT-based model, an SGD optimizer and a batch size of 35 were used, and after 7 epochs of 4 minutes each, an accuracy of 0.9676 in validation is obtained, as shown in Figure 14. An analysis of the performance of this model shows a better performance than the previous one without relevant overfitting (Figure 15).

Figure 14 Training the BERT-based model. 

Figure 15 Accuracy and Loss analysis for the BERT-based model. 

Comparison between the models

We evaluated the proposed models using a set of new issues that do not belong to the training or the validation datasets (Table 3). The results obtained by testing the different models are listed in the column titled "Probability of being a User Story." From these results, several observations are made:

The identification of short user stories improves as the complexity of the applied model increases (the BRNN-LSTM being the simplest and the BERT-based the most complex), as observed in the first example of Table 3.

Table 3 Testing new examples in the implemented models. 

Considering the User Story 5 example, the BRNN-LSTM model is not able to recognize this example as unfavorable. In contrast, others return a lower probability, ensuring this is not a positive example.

Besides orthographic errors or unknown words in a user story, all the models can generalize it correctly, as seen in the User Story 7 example.

After implementing the models and assessing their results, a comparison can be made. The metrics considered are the F1-score obtained during the validation through the library Sklearn [40], the complexity, the training time (tr-effort), the syntactic analysis (parsing), and the semantic analysis (semantic). Table 4 shows the results of the comparison.

Table 4 Comparing the models. 

As observed in Table 4, the models obtained similar values for the validation metrics; however, the most notable difference lies in the ability to semantically and syntactically analyze user stories. The BERT-based model has a slightly superior generalization capability. Besides, although the BERT-based model complexity is higher (in terms of the times and number of training epochs), it can be observed that there exists an improvement in the parsing and semantic interpretation. In contrast, the parsing of issues is similar for the BERT-based and ELMo-based models.