When we start to focus more on the technological part of BDD, especially when we talk about the concept of test automation, there is much more research done in the area often perhaps because BDD is frequently associated with test automation. It is true that the nature of Gherkin and the scenarios that are obtained in the BDD process provide an excellent basis for test automation but the truth is that BDD is much more than this. This is one of the main conclusions of the study performed by Wang and Solís[70]. The authors studied the literature to find the BDD characteristics and then seek support for them in the most commonly used in pratice BDD testing automation tools, like Cucumber and SpecFlow. One of the main conclusions reached by Wang and Solís was that these tools only supported some stages of the BDD process, especially the development phase, lacking support for the analysis phase (feature definition) but especially for the planning phase (identification of business values).
The framework comparison results can be found in fugure3.1:
The methodology used by the authors to find the BDD characteristics represented was the following: After reviewing several studies and defining an initial set of the BDD characteristics, they analysed one framework at a time using the same set, recording how the framework supported them. If they found a characteristic that was not in the list, they went back to the literature to understand if it could be considered a BDD characteristic or not.
3 . 2 . B D D P R I N C I P L E S A N D S U P P O R T I N G T O O L S
Figure 3.1: The BDD Characteristics support from seven BDD toolkits
The resulting table represents a good basis of comparison for the validation work that will be performed later in the course of this dissertation. In section3.2.1a critical overview of the results and the criteria chosen to conduct this study and produce this table will be presented.
3.2.1 Discussion on the study by Wang and Solís
The following is a critical analysis aimed at the study conducted by Wang and Solís [70], previously described. We believe that this study can be very interesting to support the validation of our work and include in it theBDDFramework and the Prototype developed
in this dissertation. However, we need to point out some things we do not agree with and we will do so by individually analyzing each of the criteria considered to draw the final table.
• Ubiquitous language definition: Although this is a central concept in Behavior- Driven Development since this language is required to express behaviors in a busi- ness speaking domain, we disagree that this process should be considered in a test automation tool. This should be considered in a different phase of the process. The definition of a ubiquitous language should be taken into account when writing tests using the Gherkin syntax, but this language must be defined elsewhere, in a more initial phase of the process. However, it can be complemented during the descrip- tion of the scenarios, since it is natural that new terms arise constantly. This is crucial to enhance communication among stakeholders and the main idea here is to run the tests without depending on an Integrated Development Environment (IDE) so that anyone involved can write the scenarios, and therefore this definition gains even more strength outside an automation framework. However, this factor can be seen as an extra, such as an auxiliary dictionary of terms, either accessible by the command line or as suggestions in an IDE, but we do not see this as essential, so much so that none of the tools studied in this investigation covers this point.
• Iterative Decomposition Process: refers to the process of identifying the features that will be part of the software product to be developed. This process begins with the identification of business outcomes. In the first place, we define the behaviors
offered by the system since these are easier to express and to analyze in terms of value. The functionalities are specified in User Stories from a User point of view and then in Gherkin scenarios. The implemented features must bring value to the client and this must be taken into account in the scheduling of the development process. Again, this is an aspect that may be outside the scope of the automation tool and it is more important in a project management tool (such asJira), where priorities among
features can be set. The responsibility of the testing framework should be only to automate and execute the features that are passed to it (already specified in Gherkin syntax), allowing, however, complete freedom regarding the execution of scenarios. The scenarios that will run in each execution must be defined as parameters. • Plain Text Description with User Story and Scenario Templates: One of the main
purposes of using such tools is to write the structured textual descriptions that constitute the specification of the functionalities (the user stories from the point of view of the user and the Gherkin Scenarios, specifying the behavior of the system) to then be interpreted and converted into test code by the automation tools. This aspect is therefore very important, especially from the point of view of the scenario definition as they will form the basis of testing documentation and automation. • Automated Acceptance Testing with Mapping Rules: It represents the main pur-
pose of the analyzed frameworks: Automated generation of test code for the defined scenarios, thus making the specifications executable. Each step will be mapped into a test method. The way this is done can vary slightly from tool to tool, and some frameworks use more or less complex regular expressions (also depending on the programming languages supported) that also allow parameter detection. Mapping rules between method names and phrases in Gherkin can also vary, as well as the structure and organization of tests in classes that test the same features or not. • Readable Behavior Oriented Specification Code: In BDD the test code should
be part of the specification of the system (living documentation) and the methods should be self-explanatory of themselves and describe the functionality so that a person that looks at the code for the first time easily realizes what it is intended to do/test. The frameworks support this by generating methods and classes with the names of the steps and with the description of the Gherkin steps, respectively. Some also support writing scenarios as code directly, with the aid of annotations. We do not agree with Wang and Solís’ rating of Cucumber on this one as it is one of the most complete tools and currently supports this, just like the rest, possibly not at the time but currently it does.
• Behavior-Driven at Different Phases: Since frameworks do not allow defining and specifying business outcomes, the authors state that the planning phase is not supported by any of the tools. However we consider that this may run away from the scope of what is really needed in the framework, being just a possible
3 . 3 . S O F T WA R E E V O LU T I O N C H A L L E N G E S
extra. In the Analysis phase, some tools support the process because they allow specifying the features through scenarios described with the Gherkin syntax and some of them even with User Stories. Finally, in the implementation phase, the same that support the analysis phase also support the automated generation of test classes and methods. The three table entries concerning this topic (planning, analysis and implementation) are somewhat repetitive and summarize what was described in the other topics, clearly dividing the process into distinct phases, which may be interesting as a conclusion of the table but not that interesting to do within it.
Some of the points taken into account in this study by Wang and Solís are considered inadequate or non-priority to be relevant in a BDD practice support test automation framework, namely the first 2, which despite their importance to the process are outside the scope of a test automation framework and as such should not be considered in this classification study. In addition some of the tools presented also have a different purpose than what is intended to be achieved during this thesis and as such will not be considered, only the two we consider most important, Cucumber and SpecFlow. After a practical analysis of some commonly used frameworks, we present a new model for evaluating BDD frameworks.